[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:02 dist-test-slave-7lbf krb5kdc[7350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:02 dist-test-slave-7lbf krb5kdc[7350](info): set up 2 sockets
Aug 10 19:19:02 dist-test-slave-7lbf krb5kdc[7350](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 19:19:03.326506 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:03 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:03.543318 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38863
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:38863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:04.282141 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:04.282665 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:04.282904 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:04.293716 7374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:04.294029 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:04.294209 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:04.294402 7374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:04.294553 7374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:04.309269 7374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:38863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:04.311838 7374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:04.314304 7374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:04.335829 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:04.335975 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:04.335927 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:04.343225 7374 server_base.cc:1034] running on GCE node
I20260810 19:19:04.344653 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:04.347436 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:04.348716 7374 hybrid_clock.cc:648] HybridClock initialized: now 1786389544348568 us; error 97 us; skew 500 ppm
I20260810 19:19:04.349440 7374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:04 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389544, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:04.354554 7374 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:04.357872 7374 webserver.cc:460] Webserver started at http://127.7.43.62:38361/ using document root <none> and password file <none>
I20260810 19:19:04.359432 7374 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:04.359697 7374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:04.360353 7374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:04.365764 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "4c53a135466a45649440139ccfc4c7de"
format_stamp: "Formatted at 2026-08-10 19:19:04 on dist-test-slave-7lbf"
server_key: "5feb009d044bf9bbe2c10b3c94840502"
server_key_iv: "6e6ec531672592b72a4293cd7156a9be"
server_key_version: "encryptionkey@0"
I20260810 19:19:04.366787 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "4c53a135466a45649440139ccfc4c7de"
format_stamp: "Formatted at 2026-08-10 19:19:04 on dist-test-slave-7lbf"
server_key: "5feb009d044bf9bbe2c10b3c94840502"
server_key_iv: "6e6ec531672592b72a4293cd7156a9be"
server_key_version: "encryptionkey@0"
I20260810 19:19:04.377419 7374 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.005s
I20260810 19:19:04.385183 7406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:04.387521 7374 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260810 19:19:04.387857 7374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "4c53a135466a45649440139ccfc4c7de"
format_stamp: "Formatted at 2026-08-10 19:19:04 on dist-test-slave-7lbf"
server_key: "5feb009d044bf9bbe2c10b3c94840502"
server_key_iv: "6e6ec531672592b72a4293cd7156a9be"
server_key_version: "encryptionkey@0"
I20260810 19:19:04.388206 7374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:04.422097 7374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:04.429105 7374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:04.429698 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:04.482049 7374 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:38863
I20260810 19:19:04.482182 7499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:38863 every 8 connection(s)
I20260810 19:19:04.485025 7374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:04.493077 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7374
I20260810 19:19:04.493604 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:04.494413 7500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:04.496510 7340 external_mini_cluster.cc:1428] Setting key 75c12ab72e61d391c8eb2116beae2f28
I20260810 19:19:04.519972 7500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de: Bootstrap starting.
I20260810 19:19:04.530010 7500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:04.533547 7500 log.cc:824] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:04.539726 7500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de: No bootstrap required, opened a new log
Aug 10 19:19:04 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:04.556077 7500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c53a135466a45649440139ccfc4c7de" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38863 } }
I20260810 19:19:04.556860 7500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:04.557170 7500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c53a135466a45649440139ccfc4c7de, State: Initialized, Role: FOLLOWER
I20260810 19:19:04.558827 7500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c53a135466a45649440139ccfc4c7de" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38863 } }
I20260810 19:19:04.559414 7500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:04.559813 7500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:04.560194 7500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:04.562552 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:04.505502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.7.43.62:38863)
0810 19:19:04.509568 (+ 4066us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:04.509590 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:04.509676 (+ 86us) server_negotiation.cc:407] Connection header received
0810 19:19:04.526107 (+ 16431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:04.526149 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:04.527101 (+ 952us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:04.527779 (+ 678us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:04.532158 (+ 4379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:04.535169 (+ 3011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:04.537956 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:04.539318 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:04.539614 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:04.548389 (+ 8775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:04.548464 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:04.548515 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:04.548587 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:04.554604 (+ 6017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:04.556076 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:04.556084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:04.556091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:04.556414 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:04.557180 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:04.557185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:04.557188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:04.558265 (+ 1077us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:04.558702 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:04.560099 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0810 19:19:04.560471 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3566,"thread_start_us":3150,"threads_started":1}
I20260810 19:19:04.564487 7500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c53a135466a45649440139ccfc4c7de" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38863 } }
I20260810 19:19:04.565658 7500 leader_election.cc:302] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c53a135466a45649440139ccfc4c7de; no voters:
I20260810 19:19:04.566984 7500 leader_election.cc:288] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:04.567317 7508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:04.569765 7508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [term 1 LEADER]: Becoming Leader. State: Replica: 4c53a135466a45649440139ccfc4c7de, State: Running, Role: LEADER
I20260810 19:19:04.570779 7508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c53a135466a45649440139ccfc4c7de" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38863 } }
I20260810 19:19:04.571059 7500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:04.579175 7509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c53a135466a45649440139ccfc4c7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c53a135466a45649440139ccfc4c7de" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38863 } } }
I20260810 19:19:04.580139 7509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:04.579689 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c53a135466a45649440139ccfc4c7de. Latest consensus state: current_term: 1 leader_uuid: "4c53a135466a45649440139ccfc4c7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c53a135466a45649440139ccfc4c7de" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38863 } } }
I20260810 19:19:04.581562 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:04.594990 7527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:04.606446 7527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:04.627717 7527 catalog_manager.cc:1347] Generated new cluster ID: 9321afc468464126a32ab7e27e83a808
I20260810 19:19:04.627882 7527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:04.649250 7527 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:04.651544 7527 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:04.663368 7527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c53a135466a45649440139ccfc4c7de: Generated new TSK 0
I20260810 19:19:04.666867 7527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:04.737501 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38863
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:05.431811 7550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:05.432188 7550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:05.432358 7550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:05.443717 7550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:05.443956 7550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:05.444169 7550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:05.458971 7550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:05.460605 7550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:05.462329 7550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:05.482131 7557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:05.482661 7562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:05.482766 7560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:05.490658 7550 server_base.cc:1034] running on GCE node
I20260810 19:19:05.491477 7550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:05.492763 7550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:05.494052 7550 hybrid_clock.cc:648] HybridClock initialized: now 1786389545493928 us; error 66 us; skew 500 ppm
I20260810 19:19:05.494514 7550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:05 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389545, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:05.499418 7550 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:05.501541 7550 webserver.cc:460] Webserver started at http://127.7.43.1:45673/ using document root <none> and password file <none>
I20260810 19:19:05.502295 7550 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:05.502404 7550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:05.502681 7550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:05.506479 7550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "5632288b38b0485b8e21e20f2eec354f"
format_stamp: "Formatted at 2026-08-10 19:19:05 on dist-test-slave-7lbf"
server_key: "b157cae67a80af9b8a9f33b17dcd8495"
server_key_iv: "6cca56bd386c94e9fe360afc475aef5e"
server_key_version: "encryptionkey@0"
I20260810 19:19:05.507476 7550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "5632288b38b0485b8e21e20f2eec354f"
format_stamp: "Formatted at 2026-08-10 19:19:05 on dist-test-slave-7lbf"
server_key: "b157cae67a80af9b8a9f33b17dcd8495"
server_key_iv: "6cca56bd386c94e9fe360afc475aef5e"
server_key_version: "encryptionkey@0"
I20260810 19:19:05.514902 7550 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260810 19:19:05.520063 7573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:05.521667 7550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:19:05.521888 7550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "5632288b38b0485b8e21e20f2eec354f"
format_stamp: "Formatted at 2026-08-10 19:19:05 on dist-test-slave-7lbf"
server_key: "b157cae67a80af9b8a9f33b17dcd8495"
server_key_iv: "6cca56bd386c94e9fe360afc475aef5e"
server_key_version: "encryptionkey@0"
I20260810 19:19:05.522233 7550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:05.542217 7550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:05.548993 7550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:05.549556 7550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:05.551515 7550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:05.554311 7550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:05.554533 7550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:05.554821 7550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:05.554980 7550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:05.647646 7550 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34711
I20260810 19:19:05.647747 7763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34711 every 8 connection(s)
I20260810 19:19:05.649456 7550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:05.659307 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7550
I20260810 19:19:05.659677 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:05.660569 7340 external_mini_cluster.cc:1428] Setting key 9b7de0cc50aa85b1a0b5199b57e7aebf
Aug 10 19:19:05 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389545, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:05.684300 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:05.654324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:41303 (local address 127.7.43.62:38863)
0810 19:19:05.655320 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:05.655325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:05.657395 (+ 2070us) server_negotiation.cc:407] Connection header received
0810 19:19:05.660916 (+ 3521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:05.660923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:05.661021 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:05.661138 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:05.664073 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:05.664824 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:05.668072 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:05.668706 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:05.668807 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:05.674989 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:05.675001 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:05.675006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:05.675051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:05.680045 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:05.681162 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:05.681167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:05.681169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:05.681380 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:05.682012 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:05.682018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:05.682020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:05.682375 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:05.682518 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:05.683369 (+ 851us) server_negotiation.cc:299] Negotiation successful
0810 19:19:05.683645 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":656,"threads_started":1}
I20260810 19:19:05.685643 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:05.655587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38863 (local address 127.7.43.1:41303)
0810 19:19:05.657063 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:05.657163 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:05.660555 (+ 3392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:05.661555 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:05.661569 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:05.662623 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:05.663829 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:05.663867 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:05.665033 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:05.665039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:05.667907 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:05.667927 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:05.669039 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:05.669049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:05.669495 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:05.670458 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:05.670493 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:05.674472 (+ 3979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:05.680266 (+ 5794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:05.680278 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:05.680307 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:05.680999 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:05.681506 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:05.681510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:05.681529 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:05.681882 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:05.682511 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:05.682520 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:05.682904 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 19:19:05.683231 (+ 327us) client_negotiation.cc:240] Negotiation successful
0810 19:19:05.683886 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":727,"threads_started":1}
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
I20260810 19:19:05.687865 7765 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38863
I20260810 19:19:05.688409 7765 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:05.689800 7765 heartbeater.cc:511] Master 127.7.43.62:38863 requested a full tablet report, sending...
I20260810 19:19:05.694207 7434 ts_manager.cc:194] Registered new tserver with Master: 5632288b38b0485b8e21e20f2eec354f (127.7.43.1:34711)
I20260810 19:19:05.696890 7434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:41303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
I20260810 19:19:06.701270 7765 heartbeater.cc:503] Master 127.7.43.62:38863 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
W20260810 19:19:06.718895 7340 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.7.43.2: real 1.055s user 0.000s sys 0.015s
I20260810 19:19:06.719843 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38863
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:07.421458 7779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:07.421931 7779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:07.422209 7779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:07.433454 7779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:07.433722 7779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:07.433974 7779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:07.448088 7779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:07.449669 7779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:07.451396 7779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:07.472147 7789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:07.472229 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:07.472291 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:07.474232 7779 server_base.cc:1034] running on GCE node
I20260810 19:19:07.475057 7779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:07.476257 7779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:07.477591 7779 hybrid_clock.cc:648] HybridClock initialized: now 1786389547477456 us; error 42 us; skew 500 ppm
I20260810 19:19:07.478091 7779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:07 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389547, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:07.486032 7779 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:07.488346 7779 webserver.cc:460] Webserver started at http://127.7.43.2:43483/ using document root <none> and password file <none>
I20260810 19:19:07.489298 7779 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:07.489502 7779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:07.489941 7779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:07.494411 7779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "b1b9f0a0f81d47b7b35850990f4ee74d"
format_stamp: "Formatted at 2026-08-10 19:19:07 on dist-test-slave-7lbf"
server_key: "87772aa99e36667fca7f107b215f6cbc"
server_key_iv: "c02ac6eacc459ed736b0b4048a5c507f"
server_key_version: "encryptionkey@0"
I20260810 19:19:07.495515 7779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "b1b9f0a0f81d47b7b35850990f4ee74d"
format_stamp: "Formatted at 2026-08-10 19:19:07 on dist-test-slave-7lbf"
server_key: "87772aa99e36667fca7f107b215f6cbc"
server_key_iv: "c02ac6eacc459ed736b0b4048a5c507f"
server_key_version: "encryptionkey@0"
I20260810 19:19:07.503248 7779 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260810 19:19:07.508956 7800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:07.510591 7779 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260810 19:19:07.510818 7779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "b1b9f0a0f81d47b7b35850990f4ee74d"
format_stamp: "Formatted at 2026-08-10 19:19:07 on dist-test-slave-7lbf"
server_key: "87772aa99e36667fca7f107b215f6cbc"
server_key_iv: "c02ac6eacc459ed736b0b4048a5c507f"
server_key_version: "encryptionkey@0"
I20260810 19:19:07.511091 7779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:07.535244 7779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:07.541744 7779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:07.542232 7779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:07.543958 7779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:07.546253 7779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:07.546394 7779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:07.546582 7779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:07.546670 7779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:07.632926 7779 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:33953
I20260810 19:19:07.633152 7984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:33953 every 8 connection(s)
I20260810 19:19:07.634657 7779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:07.640097 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7779
I20260810 19:19:07.640331 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:07.641053 7340 external_mini_cluster.cc:1428] Setting key ad5d0083b41c4c55e0553a510b754696
Aug 10 19:19:07 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389547, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:07.664391 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:07.638128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:42785 (local address 127.7.43.62:38863)
0810 19:19:07.639130 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:07.639138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:07.640158 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 19:19:07.643531 (+ 3373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:07.643536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:07.643614 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:07.643730 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:07.645924 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:07.646877 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:07.649313 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:07.649820 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:07.649959 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:07.655106 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:07.655112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:07.655115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:07.655150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:07.659315 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:07.660604 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:07.660608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:07.660610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:07.660850 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:07.661820 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:07.661826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:07.661829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:07.662332 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:07.662508 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:07.663801 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0810 19:19:07.664013 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":700,"threads_started":1}
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
I20260810 19:19:07.666179 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:07.638648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38863 (local address 127.7.43.2:42785)
0810 19:19:07.639979 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:07.640021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:07.643182 (+ 3161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:07.644023 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:07.644033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:07.644786 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:07.645742 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:07.645765 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:07.647064 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:07.647068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:07.649112 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:07.649127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:07.650070 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:07.650079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:07.650351 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:07.651128 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:07.651156 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:07.654796 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:07.659519 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:07.659532 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:07.659554 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:07.660454 (+ 900us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:07.661002 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:07.661008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:07.661012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:07.661666 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:07.662574 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:07.662586 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:07.663177 (+ 591us) client_negotiation.cc:769] Sending connection context
0810 19:19:07.663703 (+ 526us) client_negotiation.cc:240] Negotiation successful
0810 19:19:07.664423 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":727,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
I20260810 19:19:07.668068 7985 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38863
I20260810 19:19:07.668502 7985 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:07.669579 7985 heartbeater.cc:511] Master 127.7.43.62:38863 requested a full tablet report, sending...
I20260810 19:19:07.671407 7434 ts_manager.cc:194] Registered new tserver with Master: b1b9f0a0f81d47b7b35850990f4ee74d (127.7.43.2:33953)
I20260810 19:19:07.672506 7434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:42785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:07.694386 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38863
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:08.338218 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:08.338536 7993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:08.338654 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:08.349380 7993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:08.349651 7993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:08.349931 7993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:08.363905 7993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:08.365533 7993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:08.367182 7993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:08.386145 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:08.386456 8004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:08.386642 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:08.388839 7993 server_base.cc:1034] running on GCE node
I20260810 19:19:08.389667 7993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:08.390972 7993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:08.392239 7993 hybrid_clock.cc:648] HybridClock initialized: now 1786389548392132 us; error 42 us; skew 500 ppm
I20260810 19:19:08.392666 7993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:08.398294 7993 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:08.401017 7993 webserver.cc:460] Webserver started at http://127.7.43.3:41993/ using document root <none> and password file <none>
I20260810 19:19:08.402123 7993 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:08.402328 7993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:08.402768 7993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:08.406773 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "27ca23a988334cfd8a8477b0343e5d29"
format_stamp: "Formatted at 2026-08-10 19:19:08 on dist-test-slave-7lbf"
server_key: "1e87fe9f52098b8c9b63f0a285158e1c"
server_key_iv: "9081f03b637d4265b8a96f34369cad1b"
server_key_version: "encryptionkey@0"
I20260810 19:19:08.407603 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "27ca23a988334cfd8a8477b0343e5d29"
format_stamp: "Formatted at 2026-08-10 19:19:08 on dist-test-slave-7lbf"
server_key: "1e87fe9f52098b8c9b63f0a285158e1c"
server_key_iv: "9081f03b637d4265b8a96f34369cad1b"
server_key_version: "encryptionkey@0"
I20260810 19:19:08.414777 7993 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260810 19:19:08.420368 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:08.422174 7993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:19:08.422384 7993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "27ca23a988334cfd8a8477b0343e5d29"
format_stamp: "Formatted at 2026-08-10 19:19:08 on dist-test-slave-7lbf"
server_key: "1e87fe9f52098b8c9b63f0a285158e1c"
server_key_iv: "9081f03b637d4265b8a96f34369cad1b"
server_key_version: "encryptionkey@0"
I20260810 19:19:08.422674 7993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:08.474941 7993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:08.480564 7993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:08.481060 7993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:08.482770 7993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:08.485158 7993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:08.485294 7993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:08.485618 7993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:08.485803 7993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 19:19:08.566917 7993 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:46859
I20260810 19:19:08.567085 8197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:46859 every 8 connection(s)
I20260810 19:19:08.568586 7993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:08.574177 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7993
I20260810 19:19:08.574434 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:08.575091 7340 external_mini_cluster.cc:1428] Setting key 34add4b57823a1a6b149da88af3fa436
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:08.594102 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.572307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:59497 (local address 127.7.43.62:38863)
0810 19:19:08.573245 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:08.573250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:08.574381 (+ 1131us) server_negotiation.cc:407] Connection header received
0810 19:19:08.578029 (+ 3648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:08.578036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:08.578113 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:08.578210 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:08.580392 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.581592 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.584035 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.584644 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.584762 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.588327 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:08.588333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:08.588337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:08.588370 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:08.591124 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.591821 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.591827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.591829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.591959 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.592371 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.592376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.592379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.592645 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:08.592774 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:08.593469 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 19:19:08.593724 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":640,"threads_started":1}
I20260810 19:19:08.595077 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.573009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38863 (local address 127.7.43.3:59497)
0810 19:19:08.574162 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:08.574243 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:08.577708 (+ 3465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:08.578504 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:08.578513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:08.579363 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:08.580212 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:08.580234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.581782 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.581786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:08.583885 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:08.583899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.584880 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.584888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:08.585107 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.585738 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:08.585761 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:08.588077 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:08.591261 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:08.591270 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:08.591287 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:08.591707 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:08.592065 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:08.592070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:08.592074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:08.592265 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:08.592767 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:08.592774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:08.593107 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 19:19:08.593354 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 19:19:08.593947 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":594,"threads_started":1}
I20260810 19:19:08.596535 8199 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38863
I20260810 19:19:08.596861 8199 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:08.597576 8199 heartbeater.cc:511] Master 127.7.43.62:38863 requested a full tablet report, sending...
I20260810 19:19:08.598927 7434 ts_manager.cc:194] Registered new tserver with Master: 27ca23a988334cfd8a8477b0343e5d29 (127.7.43.3:46859)
I20260810 19:19:08.599723 7434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:59497
I20260810 19:19:08.604110 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:08.658216 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.640211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.7.43.1:34711)
0810 19:19:08.640881 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:08.640888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:08.640916 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:19:08.641025 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:08.641031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:08.641237 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:08.641365 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:08.642868 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.643857 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.645064 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.645711 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.645836 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.651142 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:08.651156 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:08.651172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:08.651206 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:08.655326 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.656023 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.656030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.656032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.656215 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.656534 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.656537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.656539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.656897 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:08.657196 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:08.657368 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 19:19:08.657648 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":349,"threads_started":1}
W20260810 19:19:08.659629 7617 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60710
I20260810 19:19:08.675879 7985 heartbeater.cc:503] Master 127.7.43.62:38863 was elected leader, sending a full tablet report...
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:08.712890 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.696348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.7.43.1:34711)
0810 19:19:08.696631 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:08.696639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:08.697102 (+ 463us) server_negotiation.cc:407] Connection header received
0810 19:19:08.697251 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:08.697255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:08.697322 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:08.697404 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:08.699754 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.700451 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.702215 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.702631 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.702725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.708332 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:08.708337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:08.708340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:08.708373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:08.710845 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.711410 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.711414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.711416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.711536 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.711812 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.711816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.711820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.712083 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:08.712260 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:08.712479 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 19:19:08.712648 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 19:19:08.715103 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7550
I20260810 19:19:08.731647 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.716334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.7.43.1:34711)
0810 19:19:08.716551 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:08.716556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:08.717099 (+ 543us) server_negotiation.cc:407] Connection header received
0810 19:19:08.717252 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:08.717258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:08.717317 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:08.717392 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:08.719024 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.719726 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.721400 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.721854 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.721963 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.725154 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:08.725159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:08.725162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:08.725190 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:08.728359 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.729218 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.729222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.729225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.729416 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.729889 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.729893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.729895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.730240 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:08.730549 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:08.730862 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 19:19:08.731072 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:19:08.732579 7755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:08.822667 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7550
I20260810 19:19:08.845680 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7779
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:08.862646 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.846580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.7.43.2:33953)
0810 19:19:08.847184 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:08.847191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:08.847215 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:19:08.847275 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:08.847281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:08.847495 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:08.847631 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:08.849382 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.850683 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.852367 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.853068 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.853163 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.857378 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:08.857390 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:08.857404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:08.857439 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:08.860176 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.860802 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.860809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.860811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.860929 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.861273 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.861276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.861278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.861636 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:08.861875 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:08.862138 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 19:19:08.862328 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":275,"threads_started":1}
I20260810 19:19:08.863894 7978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:08.947027 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7779
I20260810 19:19:08.969213 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7993
Aug 10 19:19:08 dist-test-slave-7lbf krb5kdc[7350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:08.985114 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:08.970271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.7.43.3:46859)
0810 19:19:08.970822 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:08.970829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:08.970859 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:19:08.970921 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:08.970936 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:08.971150 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:08.971298 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:08.972611 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.973720 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.975036 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:08.975583 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:08.975695 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:08.979752 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:08.979763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:08.979777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:08.979812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:08.982451 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.983056 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.983063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.983065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.983183 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:08.983545 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:08.983549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:08.983551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:08.983901 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:08.984193 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:08.984510 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 19:19:08.984763 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":259,"threads_started":1}
I20260810 19:19:08.986433 8190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:09.066843 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7993
I20260810 19:19:09.088160 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7374
I20260810 19:19:09.089365 7490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:09.190940 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7374
2026-08-10T19:19:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (7181 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:09 dist-test-slave-7lbf krb5kdc[8236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:09 dist-test-slave-7lbf krb5kdc[8236](info): set up 2 sockets
Aug 10 19:19:09 dist-test-slave-7lbf krb5kdc[8236](info): commencing operation
krb5kdc: starting...
W20260810 19:19:10.410871 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:10 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:10.616755 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40863
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:10.941533 8262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:10.941864 8262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:10.942006 8262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:10.949277 8262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:10.949400 8262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:10.949487 8262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:10.949586 8262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:10.949646 8262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:10.959756 8262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:10.961108 8262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:10.962427 8262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:10.974290 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:10.974208 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:10.974241 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:10.975173 8262 server_base.cc:1034] running on GCE node
I20260810 19:19:10.975973 8262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:10.977420 8262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:10.978631 8262 hybrid_clock.cc:648] HybridClock initialized: now 1786389550978582 us; error 40 us; skew 500 ppm
I20260810 19:19:10.978993 8262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:10 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:10.982007 8262 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:10.983502 8262 webserver.cc:460] Webserver started at http://127.7.43.62:34099/ using document root <none> and password file <none>
I20260810 19:19:10.984344 8262 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:10.984475 8262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:10.984786 8262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:10.987926 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "b612ae273b1941afba5eada6a6ddad3e"
format_stamp: "Formatted at 2026-08-10 19:19:10 on dist-test-slave-7lbf"
server_key: "996f04682da2935112f8b2f83096e567"
server_key_iv: "34098e19c1a71bc11b547e885e7b4a26"
server_key_version: "encryptionkey@0"
I20260810 19:19:10.988700 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "b612ae273b1941afba5eada6a6ddad3e"
format_stamp: "Formatted at 2026-08-10 19:19:10 on dist-test-slave-7lbf"
server_key: "996f04682da2935112f8b2f83096e567"
server_key_iv: "34098e19c1a71bc11b547e885e7b4a26"
server_key_version: "encryptionkey@0"
I20260810 19:19:10.994082 8262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:19:10.997707 8283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:10.999095 8262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:10.999264 8262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "b612ae273b1941afba5eada6a6ddad3e"
format_stamp: "Formatted at 2026-08-10 19:19:10 on dist-test-slave-7lbf"
server_key: "996f04682da2935112f8b2f83096e567"
server_key_iv: "34098e19c1a71bc11b547e885e7b4a26"
server_key_version: "encryptionkey@0"
I20260810 19:19:10.999455 8262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:11.022765 8262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:11.026983 8262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:11.027263 8262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:11.045020 8262 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:40863
I20260810 19:19:11.045058 8362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:40863 every 8 connection(s)
I20260810 19:19:11.046456 8262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:11.050194 8365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:11.056490 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8262
I20260810 19:19:11.056664 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:11.057291 7340 external_mini_cluster.cc:1428] Setting key b3452e420788b97b38d298d21abccf4d
I20260810 19:19:11.060753 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e: Bootstrap starting.
I20260810 19:19:11.064363 8365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:11.065536 8365 log.cc:824] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:11.068717 8365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e: No bootstrap required, opened a new log
Aug 10 19:19:11 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:11.077476 8365 raft_consensus.cc:348] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612ae273b1941afba5eada6a6ddad3e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40863 } }
I20260810 19:19:11.077888 8365 raft_consensus.cc:374] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:11.078004 8365 raft_consensus.cc:729] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b612ae273b1941afba5eada6a6ddad3e, State: Initialized, Role: FOLLOWER
I20260810 19:19:11.078699 8365 consensus_queue.cc:260] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612ae273b1941afba5eada6a6ddad3e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40863 } }
I20260810 19:19:11.078995 8365 raft_consensus.cc:388] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:11.079139 8365 raft_consensus.cc:482] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:11.079322 8365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:11.080498 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:11.059344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.7.43.62:40863)
0810 19:19:11.060160 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:11.060172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:11.060213 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:19:11.061983 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:11.062005 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:11.062412 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:11.062728 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:11.064569 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:11.066011 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:11.067518 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:11.068184 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:11.068318 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:11.072879 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:11.072887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:11.072902 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:11.072943 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:11.075974 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:11.076609 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:11.076616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:11.076623 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:11.076804 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:11.077168 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:11.077172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:11.077174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:11.077748 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:11.078023 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:11.078585 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 19:19:11.079081 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":342,"threads_started":1}
I20260810 19:19:11.082116 8365 raft_consensus.cc:504] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612ae273b1941afba5eada6a6ddad3e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40863 } }
I20260810 19:19:11.082600 8365 leader_election.cc:302] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b612ae273b1941afba5eada6a6ddad3e; no voters:
I20260810 19:19:11.083209 8365 leader_election.cc:288] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:11.083323 8372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:11.084015 8372 raft_consensus.cc:686] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [term 1 LEADER]: Becoming Leader. State: Replica: b612ae273b1941afba5eada6a6ddad3e, State: Running, Role: LEADER
I20260810 19:19:11.084554 8372 consensus_queue.cc:237] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612ae273b1941afba5eada6a6ddad3e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40863 } }
I20260810 19:19:11.085203 8365 sys_catalog.cc:564] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:11.087203 8374 sys_catalog.cc:455] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b612ae273b1941afba5eada6a6ddad3e. Latest consensus state: current_term: 1 leader_uuid: "b612ae273b1941afba5eada6a6ddad3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612ae273b1941afba5eada6a6ddad3e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40863 } } }
I20260810 19:19:11.087916 8373 sys_catalog.cc:455] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b612ae273b1941afba5eada6a6ddad3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612ae273b1941afba5eada6a6ddad3e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40863 } } }
I20260810 19:19:11.087949 8374 sys_catalog.cc:458] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:11.088335 8373 sys_catalog.cc:458] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:11.094139 8380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:11.100059 8380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:11.109476 8380 catalog_manager.cc:1347] Generated new cluster ID: 0e1df5f58991484f81251a5358a243a9
I20260810 19:19:11.109619 8380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:11.126691 8380 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:11.127991 8380 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:11.135735 8380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b612ae273b1941afba5eada6a6ddad3e: Generated new TSK 0
I20260810 19:19:11.136611 8380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:11.192464 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40863
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:11.560597 8404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:11.560861 8404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:11.560974 8404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:11.568120 8404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:11.568220 8404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:11.568341 8404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:11.577808 8404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:11.579022 8404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:11.580276 8404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:11.592695 8416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:11.592723 8419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:11.592753 8417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:11.593672 8404 server_base.cc:1034] running on GCE node
I20260810 19:19:11.594197 8404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:11.595070 8404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:11.596269 8404 hybrid_clock.cc:648] HybridClock initialized: now 1786389551596231 us; error 30 us; skew 500 ppm
I20260810 19:19:11.596639 8404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:11 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:11.599831 8404 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:11.601188 8404 webserver.cc:460] Webserver started at http://127.7.43.1:43555/ using document root <none> and password file <none>
I20260810 19:19:11.601909 8404 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:11.602033 8404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:11.602329 8404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:11.605293 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "ef21d0ff214a452b9bf3be4e4c59142b"
format_stamp: "Formatted at 2026-08-10 19:19:11 on dist-test-slave-7lbf"
server_key: "b7b2f1bca52d2c3dd55817c1db983aaf"
server_key_iv: "7e94f0f910add0f7f9f0fe98c8fb26cf"
server_key_version: "encryptionkey@0"
I20260810 19:19:11.606071 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "ef21d0ff214a452b9bf3be4e4c59142b"
format_stamp: "Formatted at 2026-08-10 19:19:11 on dist-test-slave-7lbf"
server_key: "b7b2f1bca52d2c3dd55817c1db983aaf"
server_key_iv: "7e94f0f910add0f7f9f0fe98c8fb26cf"
server_key_version: "encryptionkey@0"
I20260810 19:19:11.611014 8404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:19:11.614259 8429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:11.615510 8404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:19:11.615661 8404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "ef21d0ff214a452b9bf3be4e4c59142b"
format_stamp: "Formatted at 2026-08-10 19:19:11 on dist-test-slave-7lbf"
server_key: "b7b2f1bca52d2c3dd55817c1db983aaf"
server_key_iv: "7e94f0f910add0f7f9f0fe98c8fb26cf"
server_key_version: "encryptionkey@0"
I20260810 19:19:11.615836 8404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:11.685858 8404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:11.690474 8404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:11.690770 8404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:11.691766 8404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:11.693188 8404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:11.693269 8404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:11.693388 8404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:11.693444 8404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:11.723717 8404 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:44313
I20260810 19:19:11.723766 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:44313 every 8 connection(s)
I20260810 19:19:11.724953 8404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:11.727061 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8404
I20260810 19:19:11.727267 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:11.728080 7340 external_mini_cluster.cc:1428] Setting key 9d98db968f070617ff723debf1b21085
Aug 10 19:19:11 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:11.745266 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:11.727596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:36607 (local address 127.7.43.62:40863)
0810 19:19:11.728027 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:11.728032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:11.729243 (+ 1211us) server_negotiation.cc:407] Connection header received
0810 19:19:11.731196 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:11.731202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:11.731283 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:11.731373 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:11.732888 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:11.733654 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:11.735169 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:11.735586 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:11.735697 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:11.739175 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:11.739182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:11.739187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:11.739227 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:11.742199 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:11.742935 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:11.742941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:11.742944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:11.743115 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:11.743500 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:11.743504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:11.743507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:11.743817 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:11.743948 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:11.744660 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 19:19:11.744889 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260810 19:19:11.746068 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:11.728253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40863 (local address 127.7.43.1:36607)
0810 19:19:11.729089 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:11.729130 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:11.730936 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:11.731631 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:11.731639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:11.732153 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:11.732723 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:11.732740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:11.733805 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:11.733809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:11.735032 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:11.735045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:11.735801 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:11.735809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:11.736045 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:11.736688 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:11.736711 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:11.738946 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:11.742328 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:11.742335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:11.742348 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:11.742809 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:11.743212 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:11.743214 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:11.743217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:11.743399 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:11.743931 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:11.743936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:11.744253 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 19:19:11.744515 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 19:19:11.744914 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":317,"threads_started":1}
I20260810 19:19:11.747642 8619 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40863
I20260810 19:19:11.748001 8619 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:11.748901 8619 heartbeater.cc:511] Master 127.7.43.62:40863 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
I20260810 19:19:11.751262 8307 ts_manager.cc:194] Registered new tserver with Master: ef21d0ff214a452b9bf3be4e4c59142b (127.7.43.1:44313)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
I20260810 19:19:11.752951 8307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:36607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:11.778023 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40863
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:12.103897 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:12.104212 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:12.104363 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:12.111654 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:12.111769 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:12.111924 8632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:12.122006 8632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:12.123325 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:12.124656 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:12.137444 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:12.137488 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:12.137639 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:12.138497 8632 server_base.cc:1034] running on GCE node
I20260810 19:19:12.139103 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:12.140034 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:12.141237 8632 hybrid_clock.cc:648] HybridClock initialized: now 1786389552141184 us; error 45 us; skew 500 ppm
I20260810 19:19:12.141561 8632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:12 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:12.144616 8632 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:12.146004 8632 webserver.cc:460] Webserver started at http://127.7.43.2:41689/ using document root <none> and password file <none>
I20260810 19:19:12.146690 8632 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:12.146791 8632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:12.147056 8632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:12.149915 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "242922241ec249ab88a9c9b907defe02"
format_stamp: "Formatted at 2026-08-10 19:19:12 on dist-test-slave-7lbf"
server_key: "37e25cf289d31d00caf7365073f42895"
server_key_iv: "1e7ce2b694c0d31b6f697f7d4d33ecf8"
server_key_version: "encryptionkey@0"
I20260810 19:19:12.150678 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "242922241ec249ab88a9c9b907defe02"
format_stamp: "Formatted at 2026-08-10 19:19:12 on dist-test-slave-7lbf"
server_key: "37e25cf289d31d00caf7365073f42895"
server_key_iv: "1e7ce2b694c0d31b6f697f7d4d33ecf8"
server_key_version: "encryptionkey@0"
I20260810 19:19:12.155649 8632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:12.159135 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:12.160416 8632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:19:12.160553 8632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "242922241ec249ab88a9c9b907defe02"
format_stamp: "Formatted at 2026-08-10 19:19:12 on dist-test-slave-7lbf"
server_key: "37e25cf289d31d00caf7365073f42895"
server_key_iv: "1e7ce2b694c0d31b6f697f7d4d33ecf8"
server_key_version: "encryptionkey@0"
I20260810 19:19:12.160730 8632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:12.174257 8632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:12.177942 8632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:12.178182 8632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:12.179039 8632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:12.180270 8632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:12.180341 8632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:12.180428 8632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:12.180474 8632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:12.215862 8632 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45177
I20260810 19:19:12.216019 8856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45177 every 8 connection(s)
I20260810 19:19:12.217613 8632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:12.218037 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8632
I20260810 19:19:12.218238 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:12.218739 7340 external_mini_cluster.cc:1428] Setting key 1dc876d8a3f9372ae0dd1c7a59de02bf
Aug 10 19:19:12 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
I20260810 19:19:12.248118 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:12.221993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:60877 (local address 127.7.43.62:40863)
0810 19:19:12.222157 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:12.222161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:12.224634 (+ 2473us) server_negotiation.cc:407] Connection header received
0810 19:19:12.227562 (+ 2928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:12.227568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:12.227657 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:12.227747 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:12.229942 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:12.230881 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:12.233041 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:12.233662 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:12.233793 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:12.239546 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:12.239551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:12.239555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:12.239590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:12.244321 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:12.245310 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:12.245316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:12.245320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:12.245496 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:12.246082 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:12.246087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:12.246090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:12.246504 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:12.246664 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:12.247487 (+ 823us) server_negotiation.cc:299] Negotiation successful
0810 19:19:12.247669 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:19:12.249228 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:12.223004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40863 (local address 127.7.43.2:60877)
0810 19:19:12.224473 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:12.224515 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:12.227279 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:12.228032 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:12.228050 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:12.228816 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:12.229713 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:12.229744 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:12.231076 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:12.231082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:12.232887 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:12.232900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:12.234045 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:12.234055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:12.234507 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:12.235422 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:12.235450 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:12.239206 (+ 3756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:12.244569 (+ 5363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:12.244576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:12.244591 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:12.245129 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:12.245623 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:12.245627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:12.245629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:12.245926 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:12.246665 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:12.246674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:12.247083 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 19:19:12.247347 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 19:19:12.247804 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":802,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
I20260810 19:19:12.250844 8857 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40863
I20260810 19:19:12.251161 8857 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:12.251912 8857 heartbeater.cc:511] Master 127.7.43.62:40863 requested a full tablet report, sending...
I20260810 19:19:12.253587 8307 ts_manager.cc:194] Registered new tserver with Master: 242922241ec249ab88a9c9b907defe02 (127.7.43.2:45177)
I20260810 19:19:12.254804 8307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:60877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:12.276105 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40863
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:19:12.756625 8619 heartbeater.cc:503] Master 127.7.43.62:40863 was elected leader, sending a full tablet report...
W20260810 19:19:12.916738 8871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:12.917127 8871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:12.917337 8871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:12.927999 8871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:12.928186 8871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:12.928391 8871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:12.941751 8871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:12.943285 8871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:12.944867 8871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:12.964267 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:12.964725 8881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:12.964363 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:12.966622 8871 server_base.cc:1034] running on GCE node
I20260810 19:19:12.967501 8871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:12.968703 8871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:12.969959 8871 hybrid_clock.cc:648] HybridClock initialized: now 1786389552969863 us; error 33 us; skew 500 ppm
I20260810 19:19:12.970343 8871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:12 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:12.978960 8871 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:12.981199 8871 webserver.cc:460] Webserver started at http://127.7.43.3:41661/ using document root <none> and password file <none>
I20260810 19:19:12.982169 8871 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:12.982352 8871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:12.982726 8871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:12.986307 8871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "badf8e3add7245c2abc134848e0b26d2"
format_stamp: "Formatted at 2026-08-10 19:19:12 on dist-test-slave-7lbf"
server_key: "11b6abfc7cfcf6bce27d78db7655e3ee"
server_key_iv: "c9599d373c548c91ffc2ddf4ce7b7be9"
server_key_version: "encryptionkey@0"
I20260810 19:19:12.987262 8871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "badf8e3add7245c2abc134848e0b26d2"
format_stamp: "Formatted at 2026-08-10 19:19:12 on dist-test-slave-7lbf"
server_key: "11b6abfc7cfcf6bce27d78db7655e3ee"
server_key_iv: "c9599d373c548c91ffc2ddf4ce7b7be9"
server_key_version: "encryptionkey@0"
I20260810 19:19:12.994174 8871 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.006s
I20260810 19:19:12.999078 8896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:13.000641 8871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:19:13.000850 8871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "badf8e3add7245c2abc134848e0b26d2"
format_stamp: "Formatted at 2026-08-10 19:19:12 on dist-test-slave-7lbf"
server_key: "11b6abfc7cfcf6bce27d78db7655e3ee"
server_key_iv: "c9599d373c548c91ffc2ddf4ce7b7be9"
server_key_version: "encryptionkey@0"
I20260810 19:19:13.001160 8871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:13.032763 8871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:13.039031 8871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:13.039510 8871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:13.041188 8871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:13.043431 8871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:13.043569 8871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:13.043772 8871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:13.043880 8871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:13.120957 8871 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:34129
I20260810 19:19:13.121152 9090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:34129 every 8 connection(s)
I20260810 19:19:13.122644 8871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:13.131554 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8871
I20260810 19:19:13.131815 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:13.132735 7340 external_mini_cluster.cc:1428] Setting key 3b9c81d656d6dc96c85752f15c7fc9c4
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:13.151337 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.126168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:52383 (local address 127.7.43.62:40863)
0810 19:19:13.126941 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.126947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.128483 (+ 1536us) server_negotiation.cc:407] Connection header received
0810 19:19:13.131650 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.131656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.131750 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.131848 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.134728 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.135904 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.138203 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.138725 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.138844 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.143572 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.143577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.143580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.143613 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.147377 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.148308 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.148312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.148315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.148516 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.149116 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.149120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.149122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.149472 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.149655 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.150501 (+ 846us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.150748 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":491,"threads_started":1}
I20260810 19:19:13.152508 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.126974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40863 (local address 127.7.43.3:52383)
0810 19:19:13.128326 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:13.128366 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:13.131284 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:13.132217 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:13.132231 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:13.133429 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:13.134542 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.134570 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.136091 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.136095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.138059 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.138073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.139050 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.139060 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.139387 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.140167 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:13.140192 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:13.143323 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:13.147511 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:13.147520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:13.147534 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:13.148104 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:13.148642 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:13.148647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:13.148649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:13.148982 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:13.149622 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:13.149630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:13.150020 (+ 390us) client_negotiation.cc:769] Sending connection context
0810 19:19:13.150304 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 19:19:13.150967 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":704,"threads_started":1}
I20260810 19:19:13.154202 9092 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40863
I20260810 19:19:13.154604 9092 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:13.155582 9092 heartbeater.cc:511] Master 127.7.43.62:40863 requested a full tablet report, sending...
I20260810 19:19:13.157140 8307 ts_manager.cc:194] Registered new tserver with Master: badf8e3add7245c2abc134848e0b26d2 (127.7.43.3:34129)
I20260810 19:19:13.158156 8307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:52383
I20260810 19:19:13.161340 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:19:13.188221 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.170750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.7.43.62:40863)
0810 19:19:13.170966 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.170970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.171935 (+ 965us) server_negotiation.cc:407] Connection header received
0810 19:19:13.172130 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.172134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.172201 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.172271 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.175047 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.176002 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.177734 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.178488 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.178607 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.181874 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.181880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.181883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.181919 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.185262 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.186124 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.186128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.186131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.186299 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.186641 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.186644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.186646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.187063 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.187312 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.187630 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.187850 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:19:13.198153 8307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:13.210899 8307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:13.250784 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.232364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.7.43.1:44313)
0810 19:19:13.235216 (+ 2852us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.235226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.238738 (+ 3512us) server_negotiation.cc:407] Connection header received
0810 19:19:13.239090 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.239101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.239494 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.239696 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:13.242101 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.243745 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.247571 (+ 3826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.249182 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.249301 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.249671 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.250032 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.250230 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2632,"thread_start_us":2344,"threads_started":1}
I20260810 19:19:13.251407 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.233252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44313 (local address 127.0.0.1:51334)
0810 19:19:13.238605 (+ 5353us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:13.238624 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:13.238881 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:13.239895 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:13.239903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:13.239933 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:13.241270 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.241288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.243876 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.243882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.247356 (+ 3474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.247375 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.249402 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.249410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.249771 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.249818 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 19:19:13.249948 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 19:19:13.250136 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5182,"mutex_wait_us":42,"thread_start_us":4830,"threads_started":1}
I20260810 19:19:13.252000 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.233265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.7.43.2:45177)
0810 19:19:13.239969 (+ 6704us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.239977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.240007 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:19:13.240098 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.240105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.240381 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.240904 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:13.242454 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.244553 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.248056 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.250048 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.250175 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.250464 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.251256 (+ 792us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.251485 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6458,"thread_start_us":6276,"threads_started":1}
I20260810 19:19:13.252092 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.232395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45177 (local address 127.0.0.1:52016)
0810 19:19:13.236095 (+ 3700us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:13.236116 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:13.236269 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:13.240659 (+ 4390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:13.240665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:13.240692 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:13.242157 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.242174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.244720 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.244727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.247881 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.247894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.250758 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.250766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.251031 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.251079 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 19:19:13.251167 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:19:13.251292 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3475,"mutex_wait_us":24,"thread_start_us":3234,"threads_started":1}
I20260810 19:19:13.252799 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.231437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.7.43.3:34129)
0810 19:19:13.239387 (+ 7950us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.239396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.239453 (+ 57us) server_negotiation.cc:407] Connection header received
0810 19:19:13.239558 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.239566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.239852 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.240153 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:13.242129 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.244392 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.248525 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.250611 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.251557 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.251889 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.251967 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.252182 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7685,"thread_start_us":7286,"threads_started":1}
I20260810 19:19:13.252347 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.230897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:34129 (local address 127.0.0.1:45530)
0810 19:19:13.232120 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:13.232158 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:13.232721 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:13.240170 (+ 7449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:13.240175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:13.240202 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:13.241870 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.241884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.244568 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.244574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.248362 (+ 3788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.248378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.250960 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.250967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.251200 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.251231 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:19:13.251285 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 19:19:13.251409 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":29,"thread_start_us":669,"threads_started":1}
I20260810 19:19:13.257859 8857 heartbeater.cc:503] Master 127.7.43.62:40863 was elected leader, sending a full tablet report...
I20260810 19:19:13.257475 8510 tablet_service.cc:1467] Processing CreateTablet for tablet 690624be8b7d4dbdb8ad84380ca1bf88 (DEFAULT_TABLE table=test-table [id=18a7c4696c474435af0a460aecea756b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:13.257709 8982 tablet_service.cc:1467] Processing CreateTablet for tablet 690624be8b7d4dbdb8ad84380ca1bf88 (DEFAULT_TABLE table=test-table [id=18a7c4696c474435af0a460aecea756b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:13.260020 8510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690624be8b7d4dbdb8ad84380ca1bf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:13.260139 8982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690624be8b7d4dbdb8ad84380ca1bf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:13.260840 8727 tablet_service.cc:1467] Processing CreateTablet for tablet 690624be8b7d4dbdb8ad84380ca1bf88 (DEFAULT_TABLE table=test-table [id=18a7c4696c474435af0a460aecea756b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:13.262704 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690624be8b7d4dbdb8ad84380ca1bf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:13.277552 9127 tablet_bootstrap.cc:492] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: Bootstrap starting.
I20260810 19:19:13.282755 9130 tablet_bootstrap.cc:492] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: Bootstrap starting.
I20260810 19:19:13.283440 9127 tablet_bootstrap.cc:654] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:13.285319 9129 tablet_bootstrap.cc:492] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: Bootstrap starting.
I20260810 19:19:13.286156 9127 log.cc:824] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:13.289048 9129 tablet_bootstrap.cc:654] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:13.289076 9130 tablet_bootstrap.cc:654] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:13.290855 9127 tablet_bootstrap.cc:492] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: No bootstrap required, opened a new log
I20260810 19:19:13.290868 9129 log.cc:824] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:13.291260 9127 ts_tablet_manager.cc:1397] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260810 19:19:13.291281 9130 log.cc:824] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:13.296273 9129 tablet_bootstrap.cc:492] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: No bootstrap required, opened a new log
I20260810 19:19:13.296645 9129 ts_tablet_manager.cc:1397] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260810 19:19:13.297151 9130 tablet_bootstrap.cc:492] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: No bootstrap required, opened a new log
I20260810 19:19:13.297580 9130 ts_tablet_manager.cc:1397] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260810 19:19:13.306850 9127 raft_consensus.cc:348] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.307535 9127 raft_consensus.cc:374] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:13.307731 9127 raft_consensus.cc:729] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef21d0ff214a452b9bf3be4e4c59142b, State: Initialized, Role: FOLLOWER
I20260810 19:19:13.308557 9127 consensus_queue.cc:260] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.309592 9129 raft_consensus.cc:348] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.310295 9129 raft_consensus.cc:374] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:13.310489 9129 raft_consensus.cc:729] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badf8e3add7245c2abc134848e0b26d2, State: Initialized, Role: FOLLOWER
I20260810 19:19:13.311318 9127 ts_tablet_manager.cc:1428] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260810 19:19:13.311173 9129 consensus_queue.cc:260] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.316147 9130 raft_consensus.cc:348] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.316758 9130 raft_consensus.cc:374] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:13.316927 9130 raft_consensus.cc:729] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242922241ec249ab88a9c9b907defe02, State: Initialized, Role: FOLLOWER
I20260810 19:19:13.317873 9130 consensus_queue.cc:260] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.318162 9092 heartbeater.cc:503] Master 127.7.43.62:40863 was elected leader, sending a full tablet report...
I20260810 19:19:13.319061 9129 ts_tablet_manager.cc:1428] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: Time spent starting tablet: real 0.022s user 0.009s sys 0.014s
I20260810 19:19:13.321591 9130 ts_tablet_manager.cc:1428] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: Time spent starting tablet: real 0.024s user 0.004s sys 0.016s
W20260810 19:19:13.375540 9093 tablet.cc:2369] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:19:13.471565 8858 tablet.cc:2369] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:19:13.477779 8620 tablet.cc:2369] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:13.518440 9142 raft_consensus.cc:482] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:13.518858 9142 raft_consensus.cc:504] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.521721 9142 leader_election.cc:288] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers badf8e3add7245c2abc134848e0b26d2 (127.7.43.3:34129), ef21d0ff214a452b9bf3be4e4c59142b (127.7.43.1:44313)
I20260810 19:19:13.533771 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.522470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:58845 (local address 127.7.43.1:44313)
0810 19:19:13.522765 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.522773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.524141 (+ 1368us) server_negotiation.cc:407] Connection header received
0810 19:19:13.524380 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.524386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.524452 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.524525 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:13.525740 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.526810 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.530638 (+ 3828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.532312 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.532488 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.532617 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.533266 (+ 649us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.533412 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:19:13.533751 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.522273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:60019 (local address 127.7.43.3:34129)
0810 19:19:13.522505 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.522512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.524342 (+ 1830us) server_negotiation.cc:407] Connection header received
0810 19:19:13.524554 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.524560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.524620 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.524696 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:13.525838 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.526783 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.530612 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.532353 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.532481 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.532616 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.533262 (+ 646us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.533400 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:19:13.534425 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.522316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:34129 (local address 127.7.43.2:60019)
0810 19:19:13.524224 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:13.524239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:13.524413 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:13.524725 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:13.524729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:13.524766 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:13.525644 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.525658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.526984 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.526989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.530447 (+ 3458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.530459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.532704 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.532713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.533073 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.533137 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 19:19:13.533225 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:19:13.533443 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":93,"thread_start_us":1246,"threads_started":1}
I20260810 19:19:13.534459 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.522092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44313 (local address 127.7.43.2:58845)
0810 19:19:13.524024 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:13.524046 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:13.524239 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:13.524553 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:13.524557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:13.524580 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:13.525596 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.525611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.526931 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.526936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.530435 (+ 3499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:13.530459 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.532812 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.532819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:13.533080 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.533131 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 19:19:13.533222 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 19:19:13.533558 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"mutex_wait_us":138,"thread_start_us":1054,"threads_started":1}
I20260810 19:19:13.535971 9010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690624be8b7d4dbdb8ad84380ca1bf88" candidate_uuid: "242922241ec249ab88a9c9b907defe02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badf8e3add7245c2abc134848e0b26d2" is_pre_election: true
I20260810 19:19:13.536072 8536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690624be8b7d4dbdb8ad84380ca1bf88" candidate_uuid: "242922241ec249ab88a9c9b907defe02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" is_pre_election: true
I20260810 19:19:13.537004 9010 raft_consensus.cc:2454] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242922241ec249ab88a9c9b907defe02 in term 0.
I20260810 19:19:13.536999 8536 raft_consensus.cc:2454] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242922241ec249ab88a9c9b907defe02 in term 0.
I20260810 19:19:13.539996 8659 leader_election.cc:302] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242922241ec249ab88a9c9b907defe02, ef21d0ff214a452b9bf3be4e4c59142b; no voters:
I20260810 19:19:13.540580 9142 raft_consensus.cc:2781] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:13.540861 9142 raft_consensus.cc:482] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:13.541009 9142 raft_consensus.cc:3037] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:13.544481 9142 raft_consensus.cc:504] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.545295 9142 leader_election.cc:288] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [CANDIDATE]: Term 1 election: Requested vote from peers badf8e3add7245c2abc134848e0b26d2 (127.7.43.3:34129), ef21d0ff214a452b9bf3be4e4c59142b (127.7.43.1:44313)
I20260810 19:19:13.545943 8536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690624be8b7d4dbdb8ad84380ca1bf88" candidate_uuid: "242922241ec249ab88a9c9b907defe02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef21d0ff214a452b9bf3be4e4c59142b"
I20260810 19:19:13.545986 9010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690624be8b7d4dbdb8ad84380ca1bf88" candidate_uuid: "242922241ec249ab88a9c9b907defe02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badf8e3add7245c2abc134848e0b26d2"
I20260810 19:19:13.546244 8536 raft_consensus.cc:3037] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:13.546238 9010 raft_consensus.cc:3037] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:13.551080 9010 raft_consensus.cc:2454] T 690624be8b7d4dbdb8ad84380ca1bf88 P badf8e3add7245c2abc134848e0b26d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242922241ec249ab88a9c9b907defe02 in term 1.
I20260810 19:19:13.551141 8536 raft_consensus.cc:2454] T 690624be8b7d4dbdb8ad84380ca1bf88 P ef21d0ff214a452b9bf3be4e4c59142b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242922241ec249ab88a9c9b907defe02 in term 1.
I20260810 19:19:13.551641 8659 leader_election.cc:302] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 242922241ec249ab88a9c9b907defe02, badf8e3add7245c2abc134848e0b26d2, ef21d0ff214a452b9bf3be4e4c59142b; no voters:
I20260810 19:19:13.552042 9142 raft_consensus.cc:2781] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:13.552738 9142 raft_consensus.cc:686] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [term 1 LEADER]: Becoming Leader. State: Replica: 242922241ec249ab88a9c9b907defe02, State: Running, Role: LEADER
I20260810 19:19:13.553344 9142 consensus_queue.cc:237] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } }
I20260810 19:19:13.559648 8306 catalog_manager.cc:5495] T 690624be8b7d4dbdb8ad84380ca1bf88 P 242922241ec249ab88a9c9b907defe02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 242922241ec249ab88a9c9b907defe02 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "242922241ec249ab88a9c9b907defe02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badf8e3add7245c2abc134848e0b26d2" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242922241ec249ab88a9c9b907defe02" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef21d0ff214a452b9bf3be4e4c59142b" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44313 } health_report { overall_health: UNKNOWN } } }
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:13.625640 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.602861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.7.43.1:44313)
0810 19:19:13.603062 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.603067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.604573 (+ 1506us) server_negotiation.cc:407] Connection header received
0810 19:19:13.604785 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.604790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.604957 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.605051 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.608078 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.609302 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.611193 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.611961 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.612068 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.618039 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.618058 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.618077 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.618118 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.622407 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.623250 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.623258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.623260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.623524 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.623896 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.623899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.623901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.624402 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.624707 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.624948 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.625118 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:13.687100 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.667580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.7.43.1:44313)
0810 19:19:13.667793 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.667799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.668269 (+ 470us) server_negotiation.cc:407] Connection header received
0810 19:19:13.668460 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.668466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.668577 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.668670 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.670979 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.672316 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.674345 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.675118 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.675275 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.681037 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.681043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.681047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.681087 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.684489 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.685123 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.685127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.685129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.685313 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.685719 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.685723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.685725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.686137 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.686406 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.686513 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.686704 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:13.746578 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.727851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.7.43.1:44313)
0810 19:19:13.728093 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.728100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.728837 (+ 737us) server_negotiation.cc:407] Connection header received
0810 19:19:13.729015 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.729019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.729086 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.729168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.731724 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.732764 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.735074 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.735889 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.736018 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.739893 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.739899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.739902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.739935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.743515 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.744159 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.744163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.744165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.744327 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.744694 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.744698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.744700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.745173 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.745545 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.745792 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.746083 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 19:19:13.749099 8478 tablet_service.cc:2908] Rejecting scan request for tablet 690624be8b7d4dbdb8ad84380ca1bf88: Uninitialized: clean time has not yet been initialized
I20260810 19:19:13.752923 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8404
I20260810 19:19:13.768694 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.753780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.7.43.1:44313)
0810 19:19:13.753947 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.753951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.754324 (+ 373us) server_negotiation.cc:407] Connection header received
0810 19:19:13.754482 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.754487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.754551 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.754627 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.755770 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.756456 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.757995 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.758422 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.758597 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.761755 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.761761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.761765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.761797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.765365 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.766285 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.766289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.766292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.766443 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.766850 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.766854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.766856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.767228 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.767657 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.767957 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.768280 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:19:13.769726 8610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:13.860530 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8404
I20260810 19:19:13.888849 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8632
Aug 10 19:19:13 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:13.904675 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:13.889871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.7.43.2:45177)
0810 19:19:13.890358 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:13.890362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:13.890381 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:19:13.890442 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:13.890446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:13.890518 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:13.890634 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:13.891757 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.892470 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.893778 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:13.894185 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:13.894282 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:13.898729 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:13.898745 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:13.898763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:13.898803 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:13.901619 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.902466 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.902474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.902476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.902629 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:13.903134 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:13.903141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:13.903145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:13.903526 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:13.903792 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:13.904057 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 19:19:13.904290 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260810 19:19:13.905346 8850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:13.995157 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8632
I20260810 19:19:14.020929 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8871
Aug 10 19:19:14 dist-test-slave-7lbf krb5kdc[8236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:14.035187 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:14.021703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.7.43.3:34129)
0810 19:19:14.021889 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:14.021895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:14.021918 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:19:14.022128 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:14.022132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:14.022198 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:14.022279 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:14.023512 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:14.024221 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:14.025477 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:14.025910 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:14.026023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:14.030009 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:14.030025 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:14.030042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:14.030088 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:14.032744 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:14.033345 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:14.033352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:14.033354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:14.033509 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:14.033855 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:14.033858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:14.033860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:14.034204 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:14.034421 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:14.034620 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 19:19:14.034847 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:19:14.035820 9085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:14.122058 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8871
I20260810 19:19:14.147354 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8262
I20260810 19:19:14.148516 8355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:14.256170 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8262
2026-08-10T19:19:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5066 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:14 dist-test-slave-7lbf krb5kdc[9176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:14 dist-test-slave-7lbf krb5kdc[9176](info): set up 2 sockets
Aug 10 19:19:14 dist-test-slave-7lbf krb5kdc[9176](info): commencing operation
krb5kdc: starting...
W20260810 19:19:15.499233 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:15 dist-test-slave-7lbf krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:15.716445 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38777
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:16.044744 9194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:16.045120 9194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:16.045280 9194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:16.052981 9194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:16.053143 9194 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 19:19:16.053229 9194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:16.053332 9194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:16.053416 9194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:16.053536 9194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:16.064033 9194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:38777
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:16.065479 9194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:16.066995 9194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:16.079491 9207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:16.079501 9210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:16.079523 9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:16.080310 9194 server_base.cc:1034] running on GCE node
I20260810 19:19:16.081174 9194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:16.082664 9194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:16.083949 9194 hybrid_clock.cc:648] HybridClock initialized: now 1786389556083885 us; error 27 us; skew 500 ppm
I20260810 19:19:16.084288 9194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:16 dist-test-slave-7lbf krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389556, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:16.087383 9194 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:16.088740 9194 webserver.cc:460] Webserver started at http://127.7.43.62:33053/ using document root <none> and password file <none>
I20260810 19:19:16.089421 9194 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:16.089537 9194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:16.089870 9194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:16.092836 9194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "fe0a5296c43340b79821dacd85d29c67"
format_stamp: "Formatted at 2026-08-10 19:19:16 on dist-test-slave-7lbf"
server_key: "505d8f4194805e2d0a4bbaf826f89f53"
server_key_iv: "5426618142521d6237739c9a9d94b1ef"
server_key_version: "encryptionkey@0"
I20260810 19:19:16.093688 9194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "fe0a5296c43340b79821dacd85d29c67"
format_stamp: "Formatted at 2026-08-10 19:19:16 on dist-test-slave-7lbf"
server_key: "505d8f4194805e2d0a4bbaf826f89f53"
server_key_iv: "5426618142521d6237739c9a9d94b1ef"
server_key_version: "encryptionkey@0"
I20260810 19:19:16.098767 9194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:19:16.102249 9218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:16.103544 9194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:16.103709 9194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "fe0a5296c43340b79821dacd85d29c67"
format_stamp: "Formatted at 2026-08-10 19:19:16 on dist-test-slave-7lbf"
server_key: "505d8f4194805e2d0a4bbaf826f89f53"
server_key_iv: "5426618142521d6237739c9a9d94b1ef"
server_key_version: "encryptionkey@0"
I20260810 19:19:16.103909 9194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:16.121693 9194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:16.125841 9194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:16.126117 9194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:16.147468 9194 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:38777
I20260810 19:19:16.147480 9330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:38777 every 8 connection(s)
I20260810 19:19:16.148887 9194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:16.152734 9331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:16.158579 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9194
I20260810 19:19:16.158779 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:16.159499 7340 external_mini_cluster.cc:1428] Setting key 7a77a56bbeaa7407206190d20cd2b579
I20260810 19:19:16.164181 9331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: Bootstrap starting.
I20260810 19:19:16.168227 9331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:16.169608 9331 log.cc:824] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:16.172860 9331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: No bootstrap required, opened a new log
Aug 10 19:19:16 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:16.182310 9331 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0a5296c43340b79821dacd85d29c67" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38777 } }
I20260810 19:19:16.182816 9331 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:16.182928 9331 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a5296c43340b79821dacd85d29c67, State: Initialized, Role: FOLLOWER
I20260810 19:19:16.183687 9331 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0a5296c43340b79821dacd85d29c67" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38777 } }
I20260810 19:19:16.184006 9331 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:16.184149 9331 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:16.184331 9331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:16.184609 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:16.161747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.7.43.62:38777)
0810 19:19:16.162584 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:16.162603 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:16.162655 (+ 52us) server_negotiation.cc:407] Connection header received
0810 19:19:16.165058 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:16.165089 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:16.165654 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:16.166157 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:16.167499 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.169465 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.171117 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.172130 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.172320 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:16.176036 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:16.176047 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:16.176072 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:16.176121 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:16.180019 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:16.180690 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:16.180699 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:16.180709 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:16.180932 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:16.181252 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:16.181258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:16.181261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:16.182067 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:16.182364 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:16.182668 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 19:19:16.183141 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":356,"threads_started":1}
I20260810 19:19:16.186817 9331 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0a5296c43340b79821dacd85d29c67" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38777 } }
I20260810 19:19:16.187304 9331 leader_election.cc:302] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe0a5296c43340b79821dacd85d29c67; no voters:
I20260810 19:19:16.187844 9331 leader_election.cc:288] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:16.187964 9337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:16.188609 9337 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [term 1 LEADER]: Becoming Leader. State: Replica: fe0a5296c43340b79821dacd85d29c67, State: Running, Role: LEADER
I20260810 19:19:16.189165 9337 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0a5296c43340b79821dacd85d29c67" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38777 } }
I20260810 19:19:16.189893 9331 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:16.192431 9341 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe0a5296c43340b79821dacd85d29c67. Latest consensus state: current_term: 1 leader_uuid: "fe0a5296c43340b79821dacd85d29c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0a5296c43340b79821dacd85d29c67" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38777 } } }
I20260810 19:19:16.192405 9338 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe0a5296c43340b79821dacd85d29c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0a5296c43340b79821dacd85d29c67" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38777 } } }
I20260810 19:19:16.192849 9341 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:16.192853 9338 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:16.194566 9349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:16.200179 9349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:16.210453 9349 catalog_manager.cc:1347] Generated new cluster ID: 4cd4c935e5544115af4dfdb52cc613d9
I20260810 19:19:16.210598 9349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:16.219939 9349 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:16.221467 9349 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:16.230051 9349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: Generated new TSK 0
I20260810 19:19:16.230914 9349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:16.286830 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38777
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:16.633038 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:16.633344 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:16.633450 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:16.640898 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:16.641032 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:16.641130 9370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:19:16.641222 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:16.651520 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:16.652896 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:16.654522 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:16.668350 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:16.668263 9382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:16.668322 9380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:16.669284 9370 server_base.cc:1034] running on GCE node
I20260810 19:19:16.670051 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:16.671144 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:16.672397 9370 hybrid_clock.cc:648] HybridClock initialized: now 1786389556672317 us; error 41 us; skew 500 ppm
I20260810 19:19:16.672731 9370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:16 dist-test-slave-7lbf krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389556, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:16.676045 9370 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:16.677634 9370 webserver.cc:460] Webserver started at http://127.7.43.1:36949/ using document root <none> and password file <none>
I20260810 19:19:16.678340 9370 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:16.678427 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:16.678721 9370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:16.681931 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "91ef075e7aae45809adddf41fe68d5a5"
format_stamp: "Formatted at 2026-08-10 19:19:16 on dist-test-slave-7lbf"
server_key: "66226098eeeafc129ff7af737fead45c"
server_key_iv: "35626592f6211fe6219462685d8ffd66"
server_key_version: "encryptionkey@0"
I20260810 19:19:16.682734 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "91ef075e7aae45809adddf41fe68d5a5"
format_stamp: "Formatted at 2026-08-10 19:19:16 on dist-test-slave-7lbf"
server_key: "66226098eeeafc129ff7af737fead45c"
server_key_iv: "35626592f6211fe6219462685d8ffd66"
server_key_version: "encryptionkey@0"
I20260810 19:19:16.688082 9370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:16.691696 9391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:16.693024 9370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:19:16.693171 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "91ef075e7aae45809adddf41fe68d5a5"
format_stamp: "Formatted at 2026-08-10 19:19:16 on dist-test-slave-7lbf"
server_key: "66226098eeeafc129ff7af737fead45c"
server_key_iv: "35626592f6211fe6219462685d8ffd66"
server_key_version: "encryptionkey@0"
I20260810 19:19:16.693368 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:16.714915 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:16.719159 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:16.719411 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:16.722154 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:16.722250 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:16.722430 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:16.722504 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:19:16 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389556, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:16.747062 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:16.723094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:60487 (local address 127.7.43.62:38777)
0810 19:19:16.723578 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:16.723583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:16.724766 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 19:19:16.727007 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:16.727014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:16.727108 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:16.727234 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:16.729695 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.730723 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.733664 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.734156 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.734287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:16.739099 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:16.739107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:16.739112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:16.739150 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:16.742502 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:16.743696 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:16.743703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:16.743707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:16.743882 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:16.744633 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:16.744639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:16.744643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:16.745041 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:16.745209 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:16.746390 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0810 19:19:16.746662 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":276,"threads_started":1}
I20260810 19:19:16.748775 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:16.723748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.7.43.1:60487)
0810 19:19:16.724599 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:16.724648 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:16.726684 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:16.727618 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:16.727633 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:16.728486 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:16.729435 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:16.729463 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.730885 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.730892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:16.733396 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:16.733415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.734514 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.734528 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:16.734836 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:16.735672 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:16.735716 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:16.738786 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:16.742684 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:16.742695 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:16.742720 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:16.743539 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:16.744038 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:16.744043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:16.744051 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:16.744435 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:16.745210 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:16.745218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:16.745822 (+ 604us) client_negotiation.cc:769] Sending connection context
0810 19:19:16.746221 (+ 399us) client_negotiation.cc:240] Negotiation successful
0810 19:19:16.746903 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":341,"threads_started":1}
I20260810 19:19:16.761410 9370 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34245
I20260810 19:19:16.761493 9574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34245 every 8 connection(s)
I20260810 19:19:16.762956 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:16.765193 9575 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38777
I20260810 19:19:16.765445 9575 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:16.766235 9575 heartbeater.cc:511] Master 127.7.43.62:38777 requested a full tablet report, sending...
I20260810 19:19:16.768397 9254 ts_manager.cc:194] Registered new tserver with Master: 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245)
I20260810 19:19:16.768837 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9370
I20260810 19:19:16.769032 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:16.769935 7340 external_mini_cluster.cc:1428] Setting key 4c084ab2c4c0d638b5dd855955c0fe76
I20260810 19:19:16.770602 9254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:60487
I20260810 19:19:16.770967 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:16.756637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.7.43.1:54131)
0810 19:19:16.757363 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:16.757388 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:16.757581 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:16.757983 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:16.757990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:16.758643 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:16.759112 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:16.759127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.760505 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.760510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:16.762056 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:16.762069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.762987 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.762992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:16.763141 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:16.763795 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:16.763819 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:16.765209 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:16.768197 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:16.768201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:16.768204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:16.768566 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:16.769084 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:16.769089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:16.769093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:16.769269 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:16.769854 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:16.769858 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:16.770048 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 19:19:16.770186 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 19:19:16.770509 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":397,"threads_started":1}
I20260810 19:19:16.771014 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:16.756537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:54131 (local address 127.7.43.62:38777)
0810 19:19:16.756705 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:16.756709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:16.757494 (+ 785us) server_negotiation.cc:407] Connection header received
0810 19:19:16.757731 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:16.757738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:16.757825 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:16.757935 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:16.759323 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.760361 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.762197 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:16.762810 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:16.762955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:16.765387 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:16.765392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:16.765395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:16.765426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:16.768075 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:16.768716 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:16.768722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:16.768726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:16.768869 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:16.769395 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:16.769400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:16.769403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:16.769741 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:16.769875 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:16.770321 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 19:19:16.770555 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:16.818208 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38777
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:17.160897 9580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:17.161340 9580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:17.161545 9580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:17.170804 9580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:17.170939 9580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:17.171032 9580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:19:17.171124 9580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:17.181396 9580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:17.182804 9580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:17.184231 9580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:17.198331 9592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:17.198410 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:17.198385 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:17.199570 9580 server_base.cc:1034] running on GCE node
I20260810 19:19:17.200237 9580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:17.201260 9580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:17.202512 9580 hybrid_clock.cc:648] HybridClock initialized: now 1786389557202398 us; error 79 us; skew 500 ppm
I20260810 19:19:17.202854 9580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:17 dist-test-slave-7lbf krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389557, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:17.206298 9580 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:17.207971 9580 webserver.cc:460] Webserver started at http://127.7.43.2:36511/ using document root <none> and password file <none>
I20260810 19:19:17.208732 9580 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:17.208839 9580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:17.209185 9580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:17.212248 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "e6ec0f92ac65422b8fbd73cfea480873"
format_stamp: "Formatted at 2026-08-10 19:19:17 on dist-test-slave-7lbf"
server_key: "4691138af67f7bf4e3847b4d2c0510f1"
server_key_iv: "4e0a6426cc4390103ca55616945b7c3f"
server_key_version: "encryptionkey@0"
I20260810 19:19:17.213073 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "e6ec0f92ac65422b8fbd73cfea480873"
format_stamp: "Formatted at 2026-08-10 19:19:17 on dist-test-slave-7lbf"
server_key: "4691138af67f7bf4e3847b4d2c0510f1"
server_key_iv: "4e0a6426cc4390103ca55616945b7c3f"
server_key_version: "encryptionkey@0"
I20260810 19:19:17.218529 9580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 19:19:17.222203 9602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:17.223631 9580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:17.223796 9580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "e6ec0f92ac65422b8fbd73cfea480873"
format_stamp: "Formatted at 2026-08-10 19:19:17 on dist-test-slave-7lbf"
server_key: "4691138af67f7bf4e3847b4d2c0510f1"
server_key_iv: "4e0a6426cc4390103ca55616945b7c3f"
server_key_version: "encryptionkey@0"
I20260810 19:19:17.224028 9580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:17.239761 9580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:17.243824 9580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:17.244096 9580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:17.246999 9580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:17.247103 9580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:17.247287 9580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:17.247360 9580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:19:17 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389557, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:17.271081 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.247420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:33397 (local address 127.7.43.62:38777)
0810 19:19:17.247612 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:17.247619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:17.249114 (+ 1495us) server_negotiation.cc:407] Connection header received
0810 19:19:17.251381 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:17.251386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:17.251481 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:17.251567 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:17.253738 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.254903 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.257254 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.257981 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.258121 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.263274 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:17.263281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:17.263287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:17.263366 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:17.266794 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.267936 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.267941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.267943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.268071 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.268669 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.268675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.268678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.269108 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:17.269262 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:17.270376 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0810 19:19:17.270662 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:19:17.272135 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.248115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.7.43.2:33397)
0810 19:19:17.248961 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:17.249000 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:17.251077 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:17.251858 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:17.251871 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:17.252639 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:17.253499 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.253551 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.255094 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.255100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.257109 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.257122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.258251 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.258261 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.258575 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.259402 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:17.259450 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:17.262973 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:17.266965 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.266975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.267001 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.267782 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.268173 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.268181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.268185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.268526 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.269326 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:17.269333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:17.269882 (+ 549us) client_negotiation.cc:769] Sending connection context
0810 19:19:17.270225 (+ 343us) client_negotiation.cc:240] Negotiation successful
0810 19:19:17.270713 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":388,"threads_started":1}
I20260810 19:19:17.287159 9580 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:37571
I20260810 19:19:17.287176 9781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:37571 every 8 connection(s)
I20260810 19:19:17.288801 9580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:17.289613 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9580
I20260810 19:19:17.289793 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:17.290371 7340 external_mini_cluster.cc:1428] Setting key 6cbb39a0dc5551dec9ae5167062f3adb
I20260810 19:19:17.291144 9783 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38777
I20260810 19:19:17.291992 9783 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:17.292949 9783 heartbeater.cc:511] Master 127.7.43.62:38777 requested a full tablet report, sending...
I20260810 19:19:17.293395 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.278681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:34557 (local address 127.7.43.62:38777)
0810 19:19:17.278872 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:17.278879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:17.279461 (+ 582us) server_negotiation.cc:407] Connection header received
0810 19:19:17.279712 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:17.279718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:17.279785 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:17.279864 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:17.281130 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.282163 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.284098 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.284519 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.284630 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.286591 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:17.286599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:17.286605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:17.286646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:17.289998 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.290898 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.290905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.290909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.291073 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.291514 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.291521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.291524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.292031 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:17.292156 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:17.292783 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 19:19:17.293041 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:19:17.294539 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.278680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.7.43.2:34557)
0810 19:19:17.279354 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:17.279374 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:17.279534 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:17.279912 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:17.279917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:17.280506 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:17.280922 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.280948 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.282359 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.282368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.283964 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.283981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.284704 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.284710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.284881 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.285556 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:17.285579 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:17.286386 (+ 807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:17.290159 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.290165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.290170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.290750 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.291222 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.291227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.291231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.291387 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.292262 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:17.292268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:17.292488 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 19:19:17.293663 (+ 1175us) client_negotiation.cc:240] Negotiation successful
0810 19:19:17.294052 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":358,"threads_started":1}
I20260810 19:19:17.294755 9254 ts_manager.cc:194] Registered new tserver with Master: e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
I20260810 19:19:17.295728 9254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:33397
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:17.340585 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38777
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:19:17.774391 9575 heartbeater.cc:503] Master 127.7.43.62:38777 was elected leader, sending a full tablet report...
W20260810 19:19:17.797071 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:17.797448 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:17.797713 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:17.808529 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:17.808748 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:17.808923 9788 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:19:17.809089 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:17.822124 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:17.823639 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:17.825274 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:17.843398 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:17.843940 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:17.844013 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:17.845384 9788 server_base.cc:1034] running on GCE node
I20260810 19:19:17.846095 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:17.847096 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:17.848384 9788 hybrid_clock.cc:648] HybridClock initialized: now 1786389557848297 us; error 31 us; skew 500 ppm
I20260810 19:19:17.848841 9788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:17 dist-test-slave-7lbf krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389557, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:17.853657 9788 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:17.855612 9788 webserver.cc:460] Webserver started at http://127.7.43.3:36037/ using document root <none> and password file <none>
I20260810 19:19:17.856443 9788 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:17.856539 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:17.856840 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:17.859886 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "f271c7cf2aa6408fa5a35485363b45c5"
format_stamp: "Formatted at 2026-08-10 19:19:17 on dist-test-slave-7lbf"
server_key: "b6af1f6a6e077b294bb59af8a64bbd5d"
server_key_iv: "e096e9889913d01fb99304d373d3ec1b"
server_key_version: "encryptionkey@0"
I20260810 19:19:17.860627 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "f271c7cf2aa6408fa5a35485363b45c5"
format_stamp: "Formatted at 2026-08-10 19:19:17 on dist-test-slave-7lbf"
server_key: "b6af1f6a6e077b294bb59af8a64bbd5d"
server_key_iv: "e096e9889913d01fb99304d373d3ec1b"
server_key_version: "encryptionkey@0"
I20260810 19:19:17.867172 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260810 19:19:17.872218 9810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:17.873765 9788 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 19:19:17.874001 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "f271c7cf2aa6408fa5a35485363b45c5"
format_stamp: "Formatted at 2026-08-10 19:19:17 on dist-test-slave-7lbf"
server_key: "b6af1f6a6e077b294bb59af8a64bbd5d"
server_key_iv: "e096e9889913d01fb99304d373d3ec1b"
server_key_version: "encryptionkey@0"
I20260810 19:19:17.874367 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:17.896221 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:17.902736 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:17.903219 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:17.908560 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:17.908886 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 19:19:17.909212 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:17.909368 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:19:17 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389557, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:17.943102 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.908246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:43661 (local address 127.7.43.62:38777)
0810 19:19:17.909142 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:17.909151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:17.911150 (+ 1999us) server_negotiation.cc:407] Connection header received
0810 19:19:17.915439 (+ 4289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:17.915444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:17.915515 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:17.915610 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:17.918622 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.919981 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.923734 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.924821 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.924952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.933088 (+ 8136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:17.933099 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:17.933105 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:17.933156 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:17.937944 (+ 4788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.939319 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.939325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.939329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.939570 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.940243 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.940249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.940252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.940767 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:17.940953 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:17.942194 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0810 19:19:17.942457 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":602,"threads_started":1}
I20260810 19:19:17.944944 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.908859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.7.43.3:43661)
0810 19:19:17.910966 (+ 2107us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:17.911019 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:17.915079 (+ 4060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:17.916021 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:17.916035 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:17.917080 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:17.918389 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.918429 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.920199 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.920205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.923497 (+ 3292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.923528 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.925560 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.925571 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.925918 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.927325 (+ 1407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:17.927373 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:17.932699 (+ 5326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:17.938188 (+ 5489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.938200 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.938222 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.939138 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.939714 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.939720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.939724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.940102 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.940936 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:17.940945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:17.941565 (+ 620us) client_negotiation.cc:769] Sending connection context
0810 19:19:17.942019 (+ 454us) client_negotiation.cc:240] Negotiation successful
0810 19:19:17.942768 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"thread_start_us":1284,"threads_started":1}
I20260810 19:19:17.990155 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.959574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:49611 (local address 127.7.43.62:38777)
0810 19:19:17.959807 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:17.959813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:17.962495 (+ 2682us) server_negotiation.cc:407] Connection header received
0810 19:19:17.962900 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:17.962906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:17.962997 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:17.963082 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:17.966037 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.967506 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.970729 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.971368 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.971470 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.975637 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:17.975646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:17.975651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:17.975698 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:17.985002 (+ 9304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.986245 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.986251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.986255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.986491 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:17.987038 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:17.987045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:17.987048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:17.987534 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:17.987724 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:17.989464 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0810 19:19:17.989734 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:19:17.990861 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:17.960249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.7.43.3:49611)
0810 19:19:17.962348 (+ 2099us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:17.962374 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:17.962665 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:17.963319 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:17.963341 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:17.964573 (+ 1232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:17.965872 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.965890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.967748 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.967755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.970574 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:17.970592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:17.971931 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:17.971943 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:17.972307 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:17.973358 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:17.973386 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:17.975357 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:17.985159 (+ 9802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.985168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.985173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.986081 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.986649 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:17.986655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:17.986658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:17.986882 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:17.987805 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:17.987810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:17.988170 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 19:19:17.988361 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 19:19:17.988892 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1885,"thread_start_us":1627,"threads_started":1}
I20260810 19:19:18.011562 9788 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:43065
I20260810 19:19:18.011657 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:43065 every 8 connection(s)
I20260810 19:19:18.013188 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:18.016505 10015 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38777
I20260810 19:19:18.016781 10015 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:18.017373 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9788
I20260810 19:19:18.017669 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:18.017982 10015 heartbeater.cc:511] Master 127.7.43.62:38777 requested a full tablet report, sending...
I20260810 19:19:18.018285 7340 external_mini_cluster.cc:1428] Setting key 9c853540442d5103619fb0d28c619777
I20260810 19:19:18.019780 9254 ts_manager.cc:194] Registered new tserver with Master: f271c7cf2aa6408fa5a35485363b45c5 (127.7.43.3:43065)
I20260810 19:19:18.021026 9254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:43661
I20260810 19:19:18.021811 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:19:18 dist-test-slave-7lbf krb5kdc[9176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:19:18 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:18.082350 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.063380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.7.43.62:38777)
0810 19:19:18.063622 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.063630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.064079 (+ 449us) server_negotiation.cc:407] Connection header received
0810 19:19:18.064296 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.064301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.064369 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.064464 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:18.066917 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.067959 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.069616 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.070233 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.070349 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.075273 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:18.075279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:18.075282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:18.075319 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:18.079447 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:18.080253 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:18.080258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:18.080261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:18.080476 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:18.080844 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:18.080847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:18.080849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:18.081231 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:18.081640 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.081741 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.081933 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 19:19:18.087249 9254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:18.098485 9254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:18.131301 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.115129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:43065 (local address 127.0.0.1:53828)
0810 19:19:18.116439 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.116491 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.116669 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.119010 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.119017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.119047 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:18.119957 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.119974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.122910 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.122916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.126501 (+ 3585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.126514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.129444 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.129454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.129844 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.129916 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.130104 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.130359 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":21,"thread_start_us":828,"threads_started":1}
I20260810 19:19:18.131889 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.115705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.7.43.3:43065)
0810 19:19:18.118408 (+ 2703us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.118415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.118462 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:19:18.118545 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.118551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.118778 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.118982 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:18.120316 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.122768 (+ 2452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.126687 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.128596 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.128765 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.129101 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.131087 (+ 1986us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.131323 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2510,"thread_start_us":2162,"threads_started":1}
I20260810 19:19:18.134037 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.115750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.7.43.1:34245)
0810 19:19:18.120481 (+ 4731us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.120489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.121173 (+ 684us) server_negotiation.cc:407] Connection header received
0810 19:19:18.121379 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.121387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.121672 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.122089 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:18.123766 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.125451 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.129680 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.132142 (+ 2462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.132260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.132550 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.133379 (+ 829us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.133605 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4562,"thread_start_us":4295,"threads_started":1}
I20260810 19:19:18.134054 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.115229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34245 (local address 127.0.0.1:37846)
0810 19:19:18.120979 (+ 5750us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.121020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.121175 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.121932 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.121938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.121965 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:18.123462 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.123481 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.125777 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.125782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.129392 (+ 3610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.129407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.132860 (+ 3453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.132871 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.133117 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.133155 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.133269 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.133428 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5521,"thread_start_us":5391,"threads_started":1}
I20260810 19:19:18.137238 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.121292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.7.43.2:37571)
0810 19:19:18.122208 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.122217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.124678 (+ 2461us) server_negotiation.cc:407] Connection header received
0810 19:19:18.124817 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.124825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.125213 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.125480 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:18.126953 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.128813 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.134049 (+ 5236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.135827 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.135986 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.136234 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.136589 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.136791 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":570,"threads_started":1}
I20260810 19:19:18.137266 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.122998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:37571 (local address 127.0.0.1:50670)
0810 19:19:18.124353 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.124375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.124531 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.125450 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.125455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.125481 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:18.126604 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.126620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.129070 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.129076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.133809 (+ 4733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.133826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.136131 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.136139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.136364 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.136388 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.136462 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.136597 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":3,"thread_start_us":847,"threads_started":1}
I20260810 19:19:18.137017 9909 tablet_service.cc:1467] Processing CreateTablet for tablet f22aa77d85dd4f54958a39e702261098 (DEFAULT_TABLE table=test-table [id=2c08e869603b4b01af390dfc272591cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:18.139155 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22aa77d85dd4f54958a39e702261098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.139243 9463 tablet_service.cc:1467] Processing CreateTablet for tablet f22aa77d85dd4f54958a39e702261098 (DEFAULT_TABLE table=test-table [id=2c08e869603b4b01af390dfc272591cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:18.141290 9463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22aa77d85dd4f54958a39e702261098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.140897 9675 tablet_service.cc:1467] Processing CreateTablet for tablet f22aa77d85dd4f54958a39e702261098 (DEFAULT_TABLE table=test-table [id=2c08e869603b4b01af390dfc272591cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:18.142200 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22aa77d85dd4f54958a39e702261098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.169072 10049 tablet_bootstrap.cc:492] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Bootstrap starting.
I20260810 19:19:18.169097 10048 tablet_bootstrap.cc:492] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Bootstrap starting.
I20260810 19:19:18.175262 10050 tablet_bootstrap.cc:492] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Bootstrap starting.
I20260810 19:19:18.176509 10048 tablet_bootstrap.cc:654] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.177238 10049 tablet_bootstrap.cc:654] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.178325 10048 log.cc:824] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:18.179466 10049 log.cc:824] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:18.181053 10050 tablet_bootstrap.cc:654] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.183254 10050 log.cc:824] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:18.183341 10049 tablet_bootstrap.cc:492] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: No bootstrap required, opened a new log
I20260810 19:19:18.183630 10049 ts_tablet_manager.cc:1397] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260810 19:19:18.183957 10048 tablet_bootstrap.cc:492] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: No bootstrap required, opened a new log
I20260810 19:19:18.184378 10048 ts_tablet_manager.cc:1397] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260810 19:19:18.187503 10050 tablet_bootstrap.cc:492] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: No bootstrap required, opened a new log
I20260810 19:19:18.187867 10050 ts_tablet_manager.cc:1397] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260810 19:19:18.196081 10049 raft_consensus.cc:348] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.196666 10049 raft_consensus.cc:374] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.196833 10049 raft_consensus.cc:729] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ef075e7aae45809adddf41fe68d5a5, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.197641 10049 consensus_queue.cc:260] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.200362 10049 ts_tablet_manager.cc:1428] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260810 19:19:18.199999 10048 raft_consensus.cc:348] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.200717 10048 raft_consensus.cc:374] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.200950 10048 raft_consensus.cc:729] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ec0f92ac65422b8fbd73cfea480873, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.201773 10048 consensus_queue.cc:260] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.203524 9783 heartbeater.cc:503] Master 127.7.43.62:38777 was elected leader, sending a full tablet report...
I20260810 19:19:18.203678 10050 raft_consensus.cc:348] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.204371 10050 raft_consensus.cc:374] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.204542 10050 raft_consensus.cc:729] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f271c7cf2aa6408fa5a35485363b45c5, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.205068 10048 ts_tablet_manager.cc:1428] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Time spent starting tablet: real 0.020s user 0.011s sys 0.007s
I20260810 19:19:18.205410 10050 consensus_queue.cc:260] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.207903 10015 heartbeater.cc:503] Master 127.7.43.62:38777 was elected leader, sending a full tablet report...
I20260810 19:19:18.208245 10050 ts_tablet_manager.cc:1428] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Time spent starting tablet: real 0.020s user 0.007s sys 0.015s
W20260810 19:19:18.266103 10017 tablet.cc:2369] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:19:18.266592 9576 tablet.cc:2369] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:18.290318 10056 raft_consensus.cc:482] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:18.290935 10056 raft_consensus.cc:504] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.293784 10056 leader_election.cc:288] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245), e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
W20260810 19:19:18.297068 9784 tablet.cc:2369] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:18.303501 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.294515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:57885 (local address 127.7.43.1:34245)
0810 19:19:18.294823 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.294830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.294851 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:19:18.294945 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.294952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.295072 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.295179 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:18.296026 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.297304 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.300546 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.302274 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.302470 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.302595 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.303031 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.303168 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:19:18.303521 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.294283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34245 (local address 127.7.43.3:57885)
0810 19:19:18.294529 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.294549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.294677 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.295342 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.295347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.295377 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:18.295828 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.295843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.297473 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.297479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.300341 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.300361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.302473 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.302479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.302844 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.302883 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.302982 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.303085 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":43}
I20260810 19:19:18.304778 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22aa77d85dd4f54958a39e702261098" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ef075e7aae45809adddf41fe68d5a5" is_pre_election: true
I20260810 19:19:18.305716 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.295021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:33925 (local address 127.7.43.2:37571)
0810 19:19:18.295246 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.295251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.296420 (+ 1169us) server_negotiation.cc:407] Connection header received
0810 19:19:18.296600 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.296606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.296680 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.296760 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:18.298598 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.299795 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.303149 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.304598 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.304740 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.304879 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.305250 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.305368 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:19:18.305645 9497 raft_consensus.cc:2454] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 0.
I20260810 19:19:18.306056 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.294841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:37571 (local address 127.7.43.3:33925)
0810 19:19:18.296321 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.296336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.296475 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.296960 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.296966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.296991 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:18.298362 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.298376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.299950 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.299954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.302909 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.302922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.304925 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.304933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.305102 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.305126 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.305206 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.305330 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"mutex_wait_us":25,"thread_start_us":1099,"threads_started":1}
I20260810 19:19:18.306574 9811 leader_election.cc:302] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ef075e7aae45809adddf41fe68d5a5, f271c7cf2aa6408fa5a35485363b45c5; no voters:
I20260810 19:19:18.307097 10056 raft_consensus.cc:2781] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:18.307349 10056 raft_consensus.cc:482] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:18.307559 10056 raft_consensus.cc:3037] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.307308 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22aa77d85dd4f54958a39e702261098" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ec0f92ac65422b8fbd73cfea480873" is_pre_election: true
I20260810 19:19:18.308267 9715 raft_consensus.cc:2454] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 0.
I20260810 19:19:18.311760 10056 raft_consensus.cc:504] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.312670 10056 leader_election.cc:288] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 election: Requested vote from peers 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245), e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
I20260810 19:19:18.313138 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22aa77d85dd4f54958a39e702261098" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ec0f92ac65422b8fbd73cfea480873"
I20260810 19:19:18.313160 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22aa77d85dd4f54958a39e702261098" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ef075e7aae45809adddf41fe68d5a5"
I20260810 19:19:18.313359 9715 raft_consensus.cc:3037] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.313412 9497 raft_consensus.cc:3037] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.317438 9715 raft_consensus.cc:2454] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 1.
I20260810 19:19:18.317601 9497 raft_consensus.cc:2454] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 1.
I20260810 19:19:18.318003 9811 leader_election.cc:302] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ec0f92ac65422b8fbd73cfea480873, f271c7cf2aa6408fa5a35485363b45c5; no voters:
I20260810 19:19:18.318495 10056 raft_consensus.cc:2781] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:18.319461 10056 raft_consensus.cc:686] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 LEADER]: Becoming Leader. State: Replica: f271c7cf2aa6408fa5a35485363b45c5, State: Running, Role: LEADER
I20260810 19:19:18.320147 10056 consensus_queue.cc:237] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } }
I20260810 19:19:18.326625 9253 catalog_manager.cc:5495] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f271c7cf2aa6408fa5a35485363b45c5 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } health_report { overall_health: UNKNOWN } } }
I20260810 19:19:18.396951 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.389690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.7.43.3:43065)
0810 19:19:18.389896 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.389902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.389992 (+ 90us) server_negotiation.cc:407] Connection header received
0810 19:19:18.390402 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.390406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.390552 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.390683 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:18.391113 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.392034 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.393800 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.394525 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.394620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.395587 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:18.396248 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:18.396369 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.396519 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.396587 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:19:18.411396 9497 raft_consensus.cc:1261] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Refusing update from remote peer f271c7cf2aa6408fa5a35485363b45c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:18.412911 10059 consensus_queue.cc:1035] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 19:19:18.413604 9715 raft_consensus.cc:1261] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Refusing update from remote peer f271c7cf2aa6408fa5a35485363b45c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:18.414593 10059 consensus_queue.cc:1035] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:18.456138 9253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:18.456590 9253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:18.461102 9253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: Sending DeleteTablet for 3 replicas of tablet f22aa77d85dd4f54958a39e702261098
I20260810 19:19:18.462853 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet f22aa77d85dd4f54958a39e702261098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:18 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:53828
I20260810 19:19:18.463079 9463 tablet_service.cc:1514] Processing DeleteTablet for tablet f22aa77d85dd4f54958a39e702261098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:18 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:37846
I20260810 19:19:18.464290 9675 tablet_service.cc:1514] Processing DeleteTablet for tablet f22aa77d85dd4f54958a39e702261098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:18 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:50670
I20260810 19:19:18.467693 9253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:18.468520 9253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:18.476177 10080 tablet_replica.cc:330] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: stopping tablet replica
I20260810 19:19:18.476915 10080 raft_consensus.cc:2229] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:18.478106 10080 raft_consensus.cc:2258] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:18.481401 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 6061b7a007e745b98b49adfada3cc7bd (DEFAULT_TABLE table=test-table [id=95b46826a27542ac9a7e67851bc641db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:18.481990 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6061b7a007e745b98b49adfada3cc7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.482216 9462 tablet_service.cc:1467] Processing CreateTablet for tablet 6061b7a007e745b98b49adfada3cc7bd (DEFAULT_TABLE table=test-table [id=95b46826a27542ac9a7e67851bc641db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:18.482322 9674 tablet_service.cc:1467] Processing CreateTablet for tablet 6061b7a007e745b98b49adfada3cc7bd (DEFAULT_TABLE table=test-table [id=95b46826a27542ac9a7e67851bc641db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:18.482987 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6061b7a007e745b98b49adfada3cc7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.483201 9462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6061b7a007e745b98b49adfada3cc7bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.490804 10050 tablet_bootstrap.cc:492] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Bootstrap starting.
I20260810 19:19:18.492266 10049 tablet_bootstrap.cc:492] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Bootstrap starting.
I20260810 19:19:18.491609 10080 ts_tablet_manager.cc:1905] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:18.493808 10050 tablet_bootstrap.cc:654] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.494282 10083 tablet_replica.cc:330] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: stopping tablet replica
I20260810 19:19:18.495222 10049 tablet_bootstrap.cc:654] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.494886 10083 raft_consensus.cc:2229] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:18.497165 10083 raft_consensus.cc:2258] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:18.499845 10083 ts_tablet_manager.cc:1905] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:18.505007 10050 tablet_bootstrap.cc:492] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: No bootstrap required, opened a new log
I20260810 19:19:18.505268 10050 ts_tablet_manager.cc:1397] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260810 19:19:18.506752 10050 raft_consensus.cc:348] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.507089 10050 raft_consensus.cc:374] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.507359 10050 raft_consensus.cc:729] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f271c7cf2aa6408fa5a35485363b45c5, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.508003 10050 consensus_queue.cc:260] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.508764 10050 ts_tablet_manager.cc:1428] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 19:19:18.509483 10048 tablet_bootstrap.cc:492] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Bootstrap starting.
I20260810 19:19:18.510663 10080 ts_tablet_manager.cc:1918] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:18.511057 10080 log.cc:1196] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/f22aa77d85dd4f54958a39e702261098
I20260810 19:19:18.512132 10080 ts_tablet_manager.cc:1939] T f22aa77d85dd4f54958a39e702261098 P f271c7cf2aa6408fa5a35485363b45c5: Deleting consensus metadata
I20260810 19:19:18.512465 10048 tablet_bootstrap.cc:654] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.514595 9226 catalog_manager.cc:4841] TS f271c7cf2aa6408fa5a35485363b45c5 (127.7.43.3:43065): tablet f22aa77d85dd4f54958a39e702261098 (table test-table [id=2c08e869603b4b01af390dfc272591cd]) successfully deleted
I20260810 19:19:18.516733 10082 tablet_replica.cc:330] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: stopping tablet replica
I20260810 19:19:18.516799 10083 ts_tablet_manager.cc:1918] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:18.516974 10083 log.cc:1196] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/f22aa77d85dd4f54958a39e702261098
I20260810 19:19:18.517225 10082 raft_consensus.cc:2229] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:18.517702 10049 tablet_bootstrap.cc:492] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: No bootstrap required, opened a new log
I20260810 19:19:18.517867 10049 ts_tablet_manager.cc:1397] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.011s
I20260810 19:19:18.518013 10082 raft_consensus.cc:2258] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:18.518914 10049 raft_consensus.cc:348] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.519181 10049 raft_consensus.cc:374] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.519296 10049 raft_consensus.cc:729] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ef075e7aae45809adddf41fe68d5a5, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.519497 10083 ts_tablet_manager.cc:1939] T f22aa77d85dd4f54958a39e702261098 P e6ec0f92ac65422b8fbd73cfea480873: Deleting consensus metadata
I20260810 19:19:18.519626 10049 consensus_queue.cc:260] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.521536 9224 catalog_manager.cc:4841] TS e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571): tablet f22aa77d85dd4f54958a39e702261098 (table test-table [id=2c08e869603b4b01af390dfc272591cd]) successfully deleted
I20260810 19:19:18.522038 10049 ts_tablet_manager.cc:1428] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260810 19:19:18.522675 10048 tablet_bootstrap.cc:492] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: No bootstrap required, opened a new log
I20260810 19:19:18.522804 10048 ts_tablet_manager.cc:1397] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260810 19:19:18.522840 10082 ts_tablet_manager.cc:1905] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:18.523586 10048 raft_consensus.cc:348] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.523802 10048 raft_consensus.cc:374] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.523891 10048 raft_consensus.cc:729] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ec0f92ac65422b8fbd73cfea480873, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.524204 10048 consensus_queue.cc:260] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.525377 10048 ts_tablet_manager.cc:1428] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 19:19:18.529117 10082 ts_tablet_manager.cc:1918] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:18.529269 10082 log.cc:1196] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/f22aa77d85dd4f54958a39e702261098
I20260810 19:19:18.529871 10082 ts_tablet_manager.cc:1939] T f22aa77d85dd4f54958a39e702261098 P 91ef075e7aae45809adddf41fe68d5a5: Deleting consensus metadata
I20260810 19:19:18.531452 9224 catalog_manager.cc:4841] TS 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245): tablet f22aa77d85dd4f54958a39e702261098 (table test-table [id=2c08e869603b4b01af390dfc272591cd]) successfully deleted
I20260810 19:19:18.544024 10068 raft_consensus.cc:482] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:18.544245 10068 raft_consensus.cc:504] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.544960 10068 leader_election.cc:288] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245), e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
I20260810 19:19:18.545537 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6061b7a007e745b98b49adfada3cc7bd" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ef075e7aae45809adddf41fe68d5a5" is_pre_election: true
I20260810 19:19:18.545629 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6061b7a007e745b98b49adfada3cc7bd" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ec0f92ac65422b8fbd73cfea480873" is_pre_election: true
I20260810 19:19:18.545778 9497 raft_consensus.cc:2454] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 0.
I20260810 19:19:18.545902 9715 raft_consensus.cc:2454] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 0.
I20260810 19:19:18.546238 9811 leader_election.cc:302] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ef075e7aae45809adddf41fe68d5a5, f271c7cf2aa6408fa5a35485363b45c5; no voters:
I20260810 19:19:18.546518 10068 raft_consensus.cc:2781] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:18.546645 10068 raft_consensus.cc:482] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:18.546742 10068 raft_consensus.cc:3037] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.549988 10068 raft_consensus.cc:504] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.550691 10068 leader_election.cc:288] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 election: Requested vote from peers 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245), e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
I20260810 19:19:18.551273 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6061b7a007e745b98b49adfada3cc7bd" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ef075e7aae45809adddf41fe68d5a5"
I20260810 19:19:18.551358 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6061b7a007e745b98b49adfada3cc7bd" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ec0f92ac65422b8fbd73cfea480873"
I20260810 19:19:18.551575 9497 raft_consensus.cc:3037] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.551594 9715 raft_consensus.cc:3037] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.554787 9497 raft_consensus.cc:2454] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 1.
I20260810 19:19:18.554797 9715 raft_consensus.cc:2454] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 1.
I20260810 19:19:18.555269 9811 leader_election.cc:302] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ec0f92ac65422b8fbd73cfea480873, f271c7cf2aa6408fa5a35485363b45c5; no voters:
I20260810 19:19:18.555589 10068 raft_consensus.cc:2781] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:18.555754 10068 raft_consensus.cc:686] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 1 LEADER]: Becoming Leader. State: Replica: f271c7cf2aa6408fa5a35485363b45c5, State: Running, Role: LEADER
I20260810 19:19:18.556053 10068 consensus_queue.cc:237] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.559132 9250 catalog_manager.cc:5495] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f271c7cf2aa6408fa5a35485363b45c5 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } health_report { overall_health: HEALTHY } } }
Aug 10 19:19:18 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389556, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:18.602931 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.587785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38777 (local address 127.0.0.1:42954)
0810 19:19:18.588504 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.588527 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.588692 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.589086 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.589091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.590137 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:18.590490 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.590504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.591681 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.591687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.593228 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.593243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.593984 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.593991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.594170 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.594873 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:18.594894 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:18.597577 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:18.600508 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:18.600521 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:18.600529 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:18.600931 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:18.601273 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:18.601277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:18.601281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:18.601441 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:18.601948 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:18.601956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:18.602111 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.602250 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.602522 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":334,"threads_started":1}
I20260810 19:19:18.603206 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.587900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.7.43.62:38777)
0810 19:19:18.588793 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.588797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.588818 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:19:18.588865 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.588868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.588931 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.589019 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:18.590659 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.591549 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.593365 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.593799 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.593891 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.597763 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:18.597768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:18.597771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:18.597800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:18.600388 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:18.601040 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:18.601045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:18.601047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:18.601174 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:18.601564 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:18.601569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:18.601573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:18.601824 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:18.601960 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.602344 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.602673 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":797,"thread_start_us":680,"threads_started":1}
I20260810 19:19:18.609091 9253 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:42954:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 19:19:18.610736 9253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:18.618227 9674 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7ceb48cf514b0db82e14463c654587 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a227020cdcd647e3b8db537ce837d218]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:19:18.618211 9463 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7ceb48cf514b0db82e14463c654587 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a227020cdcd647e3b8db537ce837d218]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:19:18.618772 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7ceb48cf514b0db82e14463c654587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.618791 9463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7ceb48cf514b0db82e14463c654587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.622076 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7ceb48cf514b0db82e14463c654587 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a227020cdcd647e3b8db537ce837d218]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:19:18.622696 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7ceb48cf514b0db82e14463c654587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:18.625893 10049 tablet_bootstrap.cc:492] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: Bootstrap starting.
I20260810 19:19:18.627192 10049 tablet_bootstrap.cc:654] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.628305 10048 tablet_bootstrap.cc:492] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: Bootstrap starting.
I20260810 19:19:18.629720 10048 tablet_bootstrap.cc:654] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.630300 10050 tablet_bootstrap.cc:492] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: Bootstrap starting.
I20260810 19:19:18.631918 10048 tablet_bootstrap.cc:492] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: No bootstrap required, opened a new log
I20260810 19:19:18.632072 10050 tablet_bootstrap.cc:654] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:18.632092 10049 tablet_bootstrap.cc:492] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: No bootstrap required, opened a new log
I20260810 19:19:18.632232 10049 ts_tablet_manager.cc:1397] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260810 19:19:18.632226 10048 ts_tablet_manager.cc:1397] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260810 19:19:18.632964 10049 raft_consensus.cc:348] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.632978 10048 raft_consensus.cc:348] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.633222 10049 raft_consensus.cc:374] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.633303 10049 raft_consensus.cc:729] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ef075e7aae45809adddf41fe68d5a5, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.633345 10048 raft_consensus.cc:374] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.633625 10048 raft_consensus.cc:729] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ec0f92ac65422b8fbd73cfea480873, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.633579 10049 consensus_queue.cc:260] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.634023 10048 consensus_queue.cc:260] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.634227 10049 ts_tablet_manager.cc:1428] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:19:18.634196 10054 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.634588 10054 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.634886 10055 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.635183 10055 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.636389 10048 ts_tablet_manager.cc:1428] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260810 19:19:18.644800 10050 tablet_bootstrap.cc:492] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: No bootstrap required, opened a new log
I20260810 19:19:18.645012 10050 ts_tablet_manager.cc:1397] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.009s
I20260810 19:19:18.645884 10050 raft_consensus.cc:348] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.646188 10050 raft_consensus.cc:374] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:18.646278 10050 raft_consensus.cc:729] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f271c7cf2aa6408fa5a35485363b45c5, State: Initialized, Role: FOLLOWER
I20260810 19:19:18.646545 10050 consensus_queue.cc:260] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.647185 10050 ts_tablet_manager.cc:1428] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 19:19:18.647053 10068 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.647477 10068 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.891672 10068 raft_consensus.cc:482] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:18.891907 10068 raft_consensus.cc:504] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.892596 10068 leader_election.cc:288] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245), e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
I20260810 19:19:18.893234 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7ceb48cf514b0db82e14463c654587" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ef075e7aae45809adddf41fe68d5a5" is_pre_election: true
I20260810 19:19:18.893325 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7ceb48cf514b0db82e14463c654587" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ec0f92ac65422b8fbd73cfea480873" is_pre_election: true
I20260810 19:19:18.893569 9497 raft_consensus.cc:2454] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 0.
I20260810 19:19:18.893618 9715 raft_consensus.cc:2454] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 0.
I20260810 19:19:18.894053 9811 leader_election.cc:302] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ec0f92ac65422b8fbd73cfea480873, f271c7cf2aa6408fa5a35485363b45c5; no voters:
I20260810 19:19:18.894335 10068 raft_consensus.cc:2781] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:18.894448 10068 raft_consensus.cc:482] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:18.894560 10068 raft_consensus.cc:3037] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.896881 10068 raft_consensus.cc:504] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.897732 10068 leader_election.cc:288] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 election: Requested vote from peers 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245), e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571)
I20260810 19:19:18.898267 9497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7ceb48cf514b0db82e14463c654587" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ef075e7aae45809adddf41fe68d5a5"
I20260810 19:19:18.898362 9715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7ceb48cf514b0db82e14463c654587" candidate_uuid: "f271c7cf2aa6408fa5a35485363b45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ec0f92ac65422b8fbd73cfea480873"
I20260810 19:19:18.898494 9497 raft_consensus.cc:3037] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.898636 9715 raft_consensus.cc:3037] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:18.900521 9497 raft_consensus.cc:2454] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 1.
I20260810 19:19:18.900944 9811 leader_election.cc:302] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ef075e7aae45809adddf41fe68d5a5, f271c7cf2aa6408fa5a35485363b45c5; no voters:
I20260810 19:19:18.901199 10068 raft_consensus.cc:2781] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:18.901369 10068 raft_consensus.cc:686] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [term 1 LEADER]: Becoming Leader. State: Replica: f271c7cf2aa6408fa5a35485363b45c5, State: Running, Role: LEADER
I20260810 19:19:18.901641 10068 consensus_queue.cc:237] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } }
I20260810 19:19:18.901832 9715 raft_consensus.cc:2454] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f271c7cf2aa6408fa5a35485363b45c5 in term 1.
I20260810 19:19:18.903395 10056 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: TxnStatusTablet state changed. Reason: New leader f271c7cf2aa6408fa5a35485363b45c5. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.903679 10056 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:19:18.904405 10117 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 19:19:18.904699 10117 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 19:19:18.905189 9253 catalog_manager.cc:5495] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f271c7cf2aa6408fa5a35485363b45c5 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } health_report { overall_health: HEALTHY } } }
I20260810 19:19:18.942554 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.936672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:43065 (local address 127.0.0.1:53842)
0810 19:19:18.936932 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.936956 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.937100 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.937661 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.937664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.937674 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:18.937839 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.937846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.939302 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.939306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.940337 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.940349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.941397 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.941401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:18.941587 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.941648 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:18.942100 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:18.942124 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:19:18.942191 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 19:19:18.942316 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 19:19:18.942795 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.936724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.7.43.3:43065)
0810 19:19:18.937273 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.937291 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.937319 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:19:18.937396 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.937401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.937502 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.937648 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:18.937964 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.939174 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.940495 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.941198 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:18.941372 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:18.941823 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:18.941974 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:18.942092 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:18.942272 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 19:19:18.942384 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":269,"threads_started":1}
I20260810 19:19:18.947598 9715 raft_consensus.cc:1261] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Refusing update from remote peer f271c7cf2aa6408fa5a35485363b45c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:18.947721 9497 raft_consensus.cc:1261] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Refusing update from remote peer f271c7cf2aa6408fa5a35485363b45c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:18.948242 10056 consensus_queue.cc:1035] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:18.948580 10068 consensus_queue.cc:1035] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:18.959231 10055 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: TxnStatusTablet state changed. Reason: New leader f271c7cf2aa6408fa5a35485363b45c5. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.959553 10055 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.962802 10054 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: TxnStatusTablet state changed. Reason: New leader f271c7cf2aa6408fa5a35485363b45c5. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.963075 10054 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.963771 10121 mvcc.cc:205] Tried to move back new op lower bound from 7317051633443368960 to 7317051633263648768. Current Snapshot: MvccSnapshot[applied={T|T < 7317051633443368960}]
I20260810 19:19:18.968171 10054 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.968072 10068 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.968408 10054 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P 91ef075e7aae45809adddf41fe68d5a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.968452 10068 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:19:18.971397 10068 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.971688 10068 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P f271c7cf2aa6408fa5a35485363b45c5: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:19:18.975242 10055 tablet_replica.cc:439] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f271c7cf2aa6408fa5a35485363b45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 } } peers { permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 } } peers { permanent_uuid: "f271c7cf2aa6408fa5a35485363b45c5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43065 } } }
I20260810 19:19:18.975548 10055 tablet_replica.cc:442] T 4a7ceb48cf514b0db82e14463c654587 P e6ec0f92ac65422b8fbd73cfea480873: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:18.983994 10056 consensus_queue.cc:1035] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ec0f92ac65422b8fbd73cfea480873" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:19.001070 10056 consensus_queue.cc:1035] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ef075e7aae45809adddf41fe68d5a5" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:19.006672 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.991124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:44527 (local address 127.7.43.3:43065)
0810 19:19:18.991320 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:18.991326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:18.997419 (+ 6093us) server_negotiation.cc:407] Connection header received
0810 19:19:18.997665 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:18.997670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:18.997743 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:18.997819 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:18.998722 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:18.999948 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.003983 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.004639 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.004729 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.005959 (+ 1230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:19.006066 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:19.006141 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:19.006281 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 19:19:19.006376 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:19:19.007344 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:18.991211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:43065 (local address 127.7.43.3:44527)
0810 19:19:18.997296 (+ 6085us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:18.997318 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:18.997485 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:18.998109 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:18.998114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:18.998126 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:18.998559 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:18.998572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.001174 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.001180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:19.003844 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:19.003858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.004902 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.004908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:19.005710 (+ 802us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.005779 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:19.006160 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:19.006171 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 19:19:19.006863 (+ 692us) client_negotiation.cc:240] Negotiation successful
0810 19:19:19.007009 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5841,"thread_start_us":5685,"threads_started":1}
I20260810 19:19:19.077555 9250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:19.077808 9250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:19.080843 9250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe0a5296c43340b79821dacd85d29c67: Sending DeleteTablet for 3 replicas of tablet 6061b7a007e745b98b49adfada3cc7bd
I20260810 19:19:19.081796 9463 tablet_service.cc:1514] Processing DeleteTablet for tablet 6061b7a007e745b98b49adfada3cc7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:19 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:37846
I20260810 19:19:19.081821 9907 tablet_service.cc:1514] Processing DeleteTablet for tablet 6061b7a007e745b98b49adfada3cc7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:19 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:53828
I20260810 19:19:19.082054 9674 tablet_service.cc:1514] Processing DeleteTablet for tablet 6061b7a007e745b98b49adfada3cc7bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:19 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:50670
I20260810 19:19:19.098604 10159 tablet_replica.cc:330] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: stopping tablet replica
I20260810 19:19:19.098707 10156 tablet_replica.cc:330] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: stopping tablet replica
I20260810 19:19:19.098835 10159 raft_consensus.cc:2229] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:19.098886 10156 raft_consensus.cc:2229] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:19.099107 10156 raft_consensus.cc:2258] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:19.099226 10159 raft_consensus.cc:2258] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:19.100318 10156 ts_tablet_manager.cc:1905] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:19.104583 10159 ts_tablet_manager.cc:1905] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:19.108038 10156 ts_tablet_manager.cc:1918] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:19.108186 10156 log.cc:1196] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/6061b7a007e745b98b49adfada3cc7bd
I20260810 19:19:19.108435 10157 tablet_replica.cc:330] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: stopping tablet replica
I20260810 19:19:19.108672 10156 ts_tablet_manager.cc:1939] T 6061b7a007e745b98b49adfada3cc7bd P 91ef075e7aae45809adddf41fe68d5a5: Deleting consensus metadata
I20260810 19:19:19.108642 10157 raft_consensus.cc:2229] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:19.109088 10157 raft_consensus.cc:2258] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:19.109840 9224 catalog_manager.cc:4841] TS 91ef075e7aae45809adddf41fe68d5a5 (127.7.43.1:34245): tablet 6061b7a007e745b98b49adfada3cc7bd (table test-table [id=95b46826a27542ac9a7e67851bc641db]) successfully deleted
I20260810 19:19:19.112229 10159 ts_tablet_manager.cc:1918] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:19.112352 10159 log.cc:1196] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/6061b7a007e745b98b49adfada3cc7bd
I20260810 19:19:19.112814 10159 ts_tablet_manager.cc:1939] T 6061b7a007e745b98b49adfada3cc7bd P e6ec0f92ac65422b8fbd73cfea480873: Deleting consensus metadata
I20260810 19:19:19.113953 9224 catalog_manager.cc:4841] TS e6ec0f92ac65422b8fbd73cfea480873 (127.7.43.2:37571): tablet 6061b7a007e745b98b49adfada3cc7bd (table test-table [id=95b46826a27542ac9a7e67851bc641db]) successfully deleted
I20260810 19:19:19.114050 10157 ts_tablet_manager.cc:1905] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:19.125243 10157 ts_tablet_manager.cc:1918] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:19.125416 10157 log.cc:1196] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/6061b7a007e745b98b49adfada3cc7bd
I20260810 19:19:19.125923 10157 ts_tablet_manager.cc:1939] T 6061b7a007e745b98b49adfada3cc7bd P f271c7cf2aa6408fa5a35485363b45c5: Deleting consensus metadata
I20260810 19:19:19.127225 9226 catalog_manager.cc:4841] TS f271c7cf2aa6408fa5a35485363b45c5 (127.7.43.3:43065): tablet 6061b7a007e745b98b49adfada3cc7bd (table test-table [id=95b46826a27542ac9a7e67851bc641db]) successfully deleted
Aug 10 19:19:19 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:19.140133 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:19.109358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.7.43.1:34245)
0810 19:19:19.122160 (+ 12802us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:19.122169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:19.122914 (+ 745us) server_negotiation.cc:407] Connection header received
0810 19:19:19.123077 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:19.123083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:19.123156 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:19.123250 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:19.124765 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.125694 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.127036 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.127727 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.127858 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.132516 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:19.132539 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:19.132571 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:19.132624 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:19.137064 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.138093 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.138102 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.138104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.138255 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.138642 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.138645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.138647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.139053 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:19.139272 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:19.139631 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 19:19:19.139822 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12629,"thread_start_us":12844,"threads_started":1}
W20260810 19:19:19.141007 9567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37852
I20260810 19:19:19.161725 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:19.144019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.7.43.62:38777)
0810 19:19:19.145892 (+ 1873us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:19.145896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:19.145912 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:19:19.145962 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:19.145965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:19.146022 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:19.146114 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:19.147538 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.148683 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.150618 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.151361 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.151481 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.154422 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:19.154429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:19.154434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:19.154492 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:19.158185 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.159013 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.159019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.159023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.159165 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.159551 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.159556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.159560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.159942 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:19.160180 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:19.161089 (+ 909us) server_negotiation.cc:299] Negotiation successful
0810 19:19:19.161315 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1754,"thread_start_us":1659,"threads_started":1}
W20260810 19:19:19.162304 9250 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42964
I20260810 19:19:19.164228 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9370
I20260810 19:19:19.180810 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:19.166277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.7.43.1:34245)
0810 19:19:19.166459 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:19.166465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:19.166901 (+ 436us) server_negotiation.cc:407] Connection header received
0810 19:19:19.167156 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:19.167162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:19.167249 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:19.167355 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:19.168821 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.169970 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.171664 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.172302 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.172434 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.175041 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:19.175047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:19.175050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:19.175110 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:19.177900 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.178624 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.178631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.178634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.178844 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.179256 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.179262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.179266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.179556 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:19.179825 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:19.180194 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 19:19:19.180445 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:19:19.181439 9567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858
W20260810 19:19:19.182044 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9370: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:19.182265 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9370
I20260810 19:19:19.220168 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9580
Aug 10 19:19:19 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:19.236089 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:19.221140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.7.43.2:37571)
0810 19:19:19.221654 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:19.221658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:19.221675 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:19:19.221729 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:19.221733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:19.221794 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:19.221899 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:19.223134 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.223827 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.225080 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.225576 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.225704 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.230069 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:19.230086 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:19.230103 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:19.230152 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:19.233059 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.233930 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.233938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.233941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.234165 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.234544 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.234548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.234550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.234923 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:19.235218 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:19.235522 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 19:19:19.235747 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":247,"threads_started":1}
W20260810 19:19:19.236939 9773 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50672
W20260810 19:19:19.237715 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:19.237887 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9580
I20260810 19:19:19.269749 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9788
Aug 10 19:19:19 dist-test-slave-7lbf krb5kdc[9176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:19.288249 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:19.270803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.7.43.3:43065)
0810 19:19:19.270984 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:19.270988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:19.271038 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:19:19.271101 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:19.271105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:19.271192 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:19.271290 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:19.272987 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.273966 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.275739 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:19.276200 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:19.276304 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:19.281914 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:19.281931 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:19.281951 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:19.281989 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:19.284995 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.285926 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.285935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.285937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.286091 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:19.286525 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:19.286530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:19.286532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:19.287000 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:19.287307 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:19.287685 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:19:19.287884 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 19:19:19.289093 10005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53854
W20260810 19:19:19.289918 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9788: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:19.290098 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9788
I20260810 19:19:19.319094 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9194
I20260810 19:19:19.320492 9324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:19.437822 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9194
2026-08-10T19:19:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5182 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:19 dist-test-slave-7lbf krb5kdc[10178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:19 dist-test-slave-7lbf krb5kdc[10178](info): set up 2 sockets
Aug 10 19:19:19 dist-test-slave-7lbf krb5kdc[10178](info): commencing operation
krb5kdc: starting...
W20260810 19:19:20.662127 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:20 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:20.880554 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38253
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:38253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:21.206305 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:21.206607 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:21.206738 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:21.214054 10195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:21.214167 10195 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 19:19:21.214236 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:21.214321 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:21.214387 10195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:21.214473 10195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:21.224671 10195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:38253
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:21.226023 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:21.227355 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:21.239125 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:21.239317 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:21.239694 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:21.240054 10195 server_base.cc:1034] running on GCE node
I20260810 19:19:21.240834 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:21.242295 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:21.243520 10195 hybrid_clock.cc:648] HybridClock initialized: now 1786389561243457 us; error 37 us; skew 500 ppm
I20260810 19:19:21.243898 10195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:21 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389561, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:21.247088 10195 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:21.248549 10195 webserver.cc:460] Webserver started at http://127.7.43.62:40143/ using document root <none> and password file <none>
I20260810 19:19:21.249217 10195 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:21.249303 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:21.249625 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:21.252624 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "4e9fb43d5c504f57aedc97a8e6189631"
format_stamp: "Formatted at 2026-08-10 19:19:21 on dist-test-slave-7lbf"
server_key: "06d2c8b148841ba0b2e9f9834608d6ff"
server_key_iv: "ff31cc6d6c658ff1b360132ad31e4f08"
server_key_version: "encryptionkey@0"
I20260810 19:19:21.253356 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "4e9fb43d5c504f57aedc97a8e6189631"
format_stamp: "Formatted at 2026-08-10 19:19:21 on dist-test-slave-7lbf"
server_key: "06d2c8b148841ba0b2e9f9834608d6ff"
server_key_iv: "ff31cc6d6c658ff1b360132ad31e4f08"
server_key_version: "encryptionkey@0"
I20260810 19:19:21.258445 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 19:19:21.261854 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:21.263200 10195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:19:21.263367 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "4e9fb43d5c504f57aedc97a8e6189631"
format_stamp: "Formatted at 2026-08-10 19:19:21 on dist-test-slave-7lbf"
server_key: "06d2c8b148841ba0b2e9f9834608d6ff"
server_key_iv: "ff31cc6d6c658ff1b360132ad31e4f08"
server_key_version: "encryptionkey@0"
I20260810 19:19:21.263572 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:21.285126 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:21.289501 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:21.289856 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:21.312356 10195 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:38253
I20260810 19:19:21.312377 10330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:38253 every 8 connection(s)
I20260810 19:19:21.313910 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:21.317713 10333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:21.320753 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10195
I20260810 19:19:21.321007 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:21.321838 7340 external_mini_cluster.cc:1428] Setting key 2cf8e29b62ae318a98c3d3a96c22fcd5
I20260810 19:19:21.330673 10333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Bootstrap starting.
I20260810 19:19:21.334477 10333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:21.335850 10333 log.cc:824] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:21.338835 10333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: No bootstrap required, opened a new log
Aug 10 19:19:21 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:21.348685 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:21.324172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.7.43.62:38253)
0810 19:19:21.325097 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:21.325113 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:21.325537 (+ 424us) server_negotiation.cc:407] Connection header received
0810 19:19:21.328784 (+ 3247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:21.328817 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:21.329349 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:21.329852 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:21.331210 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.333177 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.334529 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.335595 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.335789 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:21.339886 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:21.339897 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:21.339917 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:21.339959 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:21.344019 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:21.344752 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:21.344761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:21.344770 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:21.344987 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:21.345313 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:21.345321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:21.345325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:21.346110 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:21.346417 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:21.346765 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 19:19:21.347256 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":406,"threads_started":1}
I20260810 19:19:21.349238 10333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9fb43d5c504f57aedc97a8e6189631" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38253 } }
I20260810 19:19:21.349650 10333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:21.349757 10333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9fb43d5c504f57aedc97a8e6189631, State: Initialized, Role: FOLLOWER
I20260810 19:19:21.350430 10333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9fb43d5c504f57aedc97a8e6189631" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38253 } }
I20260810 19:19:21.350756 10333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:21.350899 10333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:21.351094 10333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:21.353426 10333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9fb43d5c504f57aedc97a8e6189631" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38253 } }
I20260810 19:19:21.353937 10333 leader_election.cc:302] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e9fb43d5c504f57aedc97a8e6189631; no voters:
I20260810 19:19:21.354480 10333 leader_election.cc:288] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:21.354611 10342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:21.355294 10342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9fb43d5c504f57aedc97a8e6189631, State: Running, Role: LEADER
I20260810 19:19:21.355809 10342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9fb43d5c504f57aedc97a8e6189631" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38253 } }
I20260810 19:19:21.356489 10333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:21.359113 10344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e9fb43d5c504f57aedc97a8e6189631. Latest consensus state: current_term: 1 leader_uuid: "4e9fb43d5c504f57aedc97a8e6189631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9fb43d5c504f57aedc97a8e6189631" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38253 } } }
I20260810 19:19:21.359100 10343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e9fb43d5c504f57aedc97a8e6189631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9fb43d5c504f57aedc97a8e6189631" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38253 } } }
I20260810 19:19:21.359556 10344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:21.359557 10343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:21.360869 10351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:21.366398 10351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:21.376189 10351 catalog_manager.cc:1347] Generated new cluster ID: a97f7f66de9640de9ae9b2e88548f205
I20260810 19:19:21.376323 10351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:21.415589 10351 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:21.416996 10351 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:21.432559 10351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Generated new TSK 0
I20260810 19:19:21.433724 10351 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:21.501252 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38253
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:21.834926 10370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:21.835215 10370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:21.835363 10370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:21.842651 10370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:21.842763 10370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:21.842885 10370 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 19:19:21.842945 10370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:19:21.843034 10370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:21.853098 10370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:38253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:21.854483 10370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:21.855809 10370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:21.869006 10379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:21.869110 10378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:21.869086 10381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:21.870102 10370 server_base.cc:1034] running on GCE node
I20260810 19:19:21.870713 10370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:21.871693 10370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:21.872898 10370 hybrid_clock.cc:648] HybridClock initialized: now 1786389561872842 us; error 47 us; skew 500 ppm
I20260810 19:19:21.873221 10370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:21 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389561, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:21.876441 10370 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:21.877954 10370 webserver.cc:460] Webserver started at http://127.7.43.1:40907/ using document root <none> and password file <none>
I20260810 19:19:21.878662 10370 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:21.878779 10370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:21.879096 10370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:21.882172 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "5899515cff904a8397fa4260be94ebc9"
format_stamp: "Formatted at 2026-08-10 19:19:21 on dist-test-slave-7lbf"
server_key: "a443ece53e61d6a7c0a2303bf4bacae5"
server_key_iv: "061e879e0690c45ad7827c1d98d3c7c7"
server_key_version: "encryptionkey@0"
I20260810 19:19:21.882972 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "5899515cff904a8397fa4260be94ebc9"
format_stamp: "Formatted at 2026-08-10 19:19:21 on dist-test-slave-7lbf"
server_key: "a443ece53e61d6a7c0a2303bf4bacae5"
server_key_iv: "061e879e0690c45ad7827c1d98d3c7c7"
server_key_version: "encryptionkey@0"
I20260810 19:19:21.888437 10370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:19:21.892118 10395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:21.893821 10370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:21.893994 10370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "5899515cff904a8397fa4260be94ebc9"
format_stamp: "Formatted at 2026-08-10 19:19:21 on dist-test-slave-7lbf"
server_key: "a443ece53e61d6a7c0a2303bf4bacae5"
server_key_iv: "061e879e0690c45ad7827c1d98d3c7c7"
server_key_version: "encryptionkey@0"
I20260810 19:19:21.894213 10370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:21.917299 10370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:21.921397 10370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:21.921694 10370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:21.924388 10370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:21.924500 10370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:21.924623 10370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:21.924707 10370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:19:21 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389561, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:21.948683 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:21.925229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:53077 (local address 127.7.43.62:38253)
0810 19:19:21.925721 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:21.925726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:21.927084 (+ 1358us) server_negotiation.cc:407] Connection header received
0810 19:19:21.929346 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:21.929353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:21.929450 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:21.929614 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:21.931858 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.932855 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.935690 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.936294 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.936424 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:21.941675 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:21.941683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:21.941688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:21.941734 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:21.945445 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:21.946214 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:21.946220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:21.946224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:21.946400 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:21.946900 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:21.946904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:21.946907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:21.947213 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:21.947353 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:21.948101 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 19:19:21.948329 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":280,"threads_started":1}
I20260810 19:19:21.949790 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:21.925955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.7.43.1:53077)
0810 19:19:21.926797 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:21.926883 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:21.929044 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:21.929877 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:21.929891 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:21.930758 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:21.931636 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:21.931659 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.933044 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.933048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:21.935448 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:21.935470 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.936622 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.936636 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:21.937061 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:21.938008 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:21.938054 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:21.941370 (+ 3316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:21.945631 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:21.945638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:21.945651 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:21.946089 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:21.946531 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:21.946536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:21.946539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:21.946763 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:21.947348 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:21.947356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:21.947721 (+ 365us) client_negotiation.cc:769] Sending connection context
0810 19:19:21.947961 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 19:19:21.948495 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":365,"threads_started":1}
I20260810 19:19:21.968986 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:21.955778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.7.43.1:49425)
0810 19:19:21.956420 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:21.956439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:21.956578 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:21.957049 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:21.957058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:21.957856 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:21.958215 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:21.958225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.959429 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.959433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:21.961031 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:21.961046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.961992 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.962000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:21.962158 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:21.962718 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:21.962735 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:21.963420 (+ 685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:21.966277 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:21.966282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:21.966284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:21.966908 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:21.967343 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:21.967349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:21.967352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:21.967499 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:21.968089 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:21.968092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:21.968253 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 19:19:21.968390 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 19:19:21.968629 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":338,"threads_started":1}
I20260810 19:19:21.969106 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:21.955750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:49425 (local address 127.7.43.62:38253)
0810 19:19:21.955898 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:21.955903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:21.956512 (+ 609us) server_negotiation.cc:407] Connection header received
0810 19:19:21.956688 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:21.956694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:21.956774 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:21.956862 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:21.958353 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.959259 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.961209 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:21.961749 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:21.961859 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:21.963613 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:21.963618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:21.963621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:21.963652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:21.966137 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:21.967028 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:21.967033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:21.967036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:21.967209 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:21.967608 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:21.967613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:21.967617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:21.967992 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:21.968140 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:21.968519 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 19:19:21.968751 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:19:21.976019 10370 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34341
I20260810 19:19:21.976043 10586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34341 every 8 connection(s)
I20260810 19:19:21.977268 10370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:21.979281 10590 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38253
I20260810 19:19:21.979527 10590 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:21.980154 10590 heartbeater.cc:511] Master 127.7.43.62:38253 requested a full tablet report, sending...
I20260810 19:19:21.982153 10245 ts_manager.cc:194] Registered new tserver with Master: 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341)
I20260810 19:19:21.983567 10245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:53077
I20260810 19:19:21.985116 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10370
I20260810 19:19:21.985340 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:21.986160 7340 external_mini_cluster.cc:1428] Setting key 8e69c6cf144bfc8dea881a11de90e0cf
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:22.036079 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38253
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:22.359505 10595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:22.359786 10595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:22.359927 10595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:22.367255 10595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:22.367374 10595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:22.367514 10595 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 19:19:22.367576 10595 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:19:22.367645 10595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:22.377544 10595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:38253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:22.378808 10595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:22.380170 10595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:22.393535 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:22.393440 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:22.393482 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:22.394541 10595 server_base.cc:1034] running on GCE node
I20260810 19:19:22.395285 10595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:22.396240 10595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:22.397470 10595 hybrid_clock.cc:648] HybridClock initialized: now 1786389562397408 us; error 45 us; skew 500 ppm
I20260810 19:19:22.397888 10595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:22 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389562, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:22.401080 10595 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:22.402604 10595 webserver.cc:460] Webserver started at http://127.7.43.2:44949/ using document root <none> and password file <none>
I20260810 19:19:22.403321 10595 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:22.403445 10595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:22.403751 10595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:22.406837 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "07d2627090cd49788c1bbacd8dce32a3"
format_stamp: "Formatted at 2026-08-10 19:19:22 on dist-test-slave-7lbf"
server_key: "8265dafbad7ecb609abd7338c8b8e98f"
server_key_iv: "ec66ae943d4e96aecc2052ea0470e90c"
server_key_version: "encryptionkey@0"
I20260810 19:19:22.407599 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "07d2627090cd49788c1bbacd8dce32a3"
format_stamp: "Formatted at 2026-08-10 19:19:22 on dist-test-slave-7lbf"
server_key: "8265dafbad7ecb609abd7338c8b8e98f"
server_key_iv: "ec66ae943d4e96aecc2052ea0470e90c"
server_key_version: "encryptionkey@0"
I20260810 19:19:22.413013 10595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:19:22.416780 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:22.418164 10595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:22.418318 10595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "07d2627090cd49788c1bbacd8dce32a3"
format_stamp: "Formatted at 2026-08-10 19:19:22 on dist-test-slave-7lbf"
server_key: "8265dafbad7ecb609abd7338c8b8e98f"
server_key_iv: "ec66ae943d4e96aecc2052ea0470e90c"
server_key_version: "encryptionkey@0"
I20260810 19:19:22.418526 10595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:22.441653 10595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:22.445425 10595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:22.445739 10595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:22.448364 10595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:22.448482 10595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:22.448598 10595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:22.448688 10595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:19:22 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389562, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:22.476354 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.449137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:33933 (local address 127.7.43.62:38253)
0810 19:19:22.449331 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:22.449338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:22.450893 (+ 1555us) server_negotiation.cc:407] Connection header received
0810 19:19:22.453833 (+ 2940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:22.453840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:22.453936 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:22.454037 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:22.456954 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.458151 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.461189 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.461963 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.462113 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:22.468354 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:22.468361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:22.468366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:22.468417 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:22.472166 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:22.473364 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:22.473372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:22.473375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:22.473604 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:22.474166 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:22.474173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:22.474177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:22.474520 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:22.474670 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:22.475661 (+ 991us) server_negotiation.cc:299] Negotiation successful
0810 19:19:22.475936 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:19:22.477547 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.449711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.7.43.2:33933)
0810 19:19:22.450684 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:22.450755 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:22.453400 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:22.454448 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:22.454459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:22.455488 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:22.456686 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.456722 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.458357 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.458363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.460987 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.461008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.462609 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.462621 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.463000 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:22.463860 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:22.463896 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:22.467971 (+ 4075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:22.472374 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:22.472388 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:22.472408 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:22.473199 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:22.473775 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:22.473779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:22.473783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:22.474031 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:22.474680 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:22.474689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:22.475199 (+ 510us) client_negotiation.cc:769] Sending connection context
0810 19:19:22.475507 (+ 308us) client_negotiation.cc:240] Negotiation successful
0810 19:19:22.476103 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":359,"threads_started":1}
I20260810 19:19:22.497390 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.484189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:55225 (local address 127.7.43.62:38253)
0810 19:19:22.484384 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:22.484390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:22.484514 (+ 124us) server_negotiation.cc:407] Connection header received
0810 19:19:22.484705 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:22.484709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:22.484780 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:22.484864 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:22.486035 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.487088 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.488651 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.489199 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.489325 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:22.491884 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:22.491891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:22.491894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:22.491934 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:22.494499 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:22.495412 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:22.495418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:22.495422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:22.495589 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:22.495927 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:22.495932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:22.495936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:22.496279 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:22.496415 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:22.496830 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 19:19:22.497065 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:19:22.497705 10595 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:34427
I20260810 19:19:22.497810 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.483757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.7.43.2:55225)
0810 19:19:22.484420 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:22.484439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:22.484587 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:22.484908 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:22.484912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:22.485391 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:22.485854 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.485872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.487219 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.487225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.488511 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.488527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.489656 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.489662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.489803 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:22.490555 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:22.490574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:22.491701 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:22.494626 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:22.494632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:22.494639 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:22.495243 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:22.495687 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:22.495692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:22.495696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:22.495822 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:22.496397 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:22.496402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:22.496592 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 19:19:22.496722 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 19:19:22.497143 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":489,"threads_started":1}
I20260810 19:19:22.497963 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:34427 every 8 connection(s)
I20260810 19:19:22.499331 10595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:22.501355 10809 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38253
I20260810 19:19:22.501609 10809 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:22.502290 10809 heartbeater.cc:511] Master 127.7.43.62:38253 requested a full tablet report, sending...
I20260810 19:19:22.503585 10245 ts_manager.cc:194] Registered new tserver with Master: 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427)
I20260810 19:19:22.504348 10245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:33933
I20260810 19:19:22.508035 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10595
I20260810 19:19:22.508193 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:22.508556 7340 external_mini_cluster.cc:1428] Setting key a84ff0d18754e14ab0975912e292c3a5
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:22.557301 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38253
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:22.885455 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:22.885768 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:22.885918 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:22.893158 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:22.893285 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:22.893410 10816 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 19:19:22.893474 10816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:19:22.893601 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:22.903664 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:38253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:22.904995 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:22.906342 10816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:22.919014 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:22.919080 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:22.919023 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:22.919672 10816 server_base.cc:1034] running on GCE node
I20260810 19:19:22.920306 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:22.921213 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:22.922417 10816 hybrid_clock.cc:648] HybridClock initialized: now 1786389562922360 us; error 43 us; skew 500 ppm
I20260810 19:19:22.922756 10816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:22 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389562, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:22.925946 10816 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:22.927439 10816 webserver.cc:460] Webserver started at http://127.7.43.3:45887/ using document root <none> and password file <none>
I20260810 19:19:22.928113 10816 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:22.928229 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:22.928534 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:22.931511 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "4a41b69697184a378c82e6ae3cb16a03"
format_stamp: "Formatted at 2026-08-10 19:19:22 on dist-test-slave-7lbf"
server_key: "c3ea77227f1f66ed2f4da52cb97e6d3c"
server_key_iv: "605bcfc036687d9d90528d4ea2b90330"
server_key_version: "encryptionkey@0"
I20260810 19:19:22.932256 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "4a41b69697184a378c82e6ae3cb16a03"
format_stamp: "Formatted at 2026-08-10 19:19:22 on dist-test-slave-7lbf"
server_key: "c3ea77227f1f66ed2f4da52cb97e6d3c"
server_key_iv: "605bcfc036687d9d90528d4ea2b90330"
server_key_version: "encryptionkey@0"
I20260810 19:19:22.937594 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:19:22.941182 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:22.942596 10816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:19:22.942756 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "4a41b69697184a378c82e6ae3cb16a03"
format_stamp: "Formatted at 2026-08-10 19:19:22 on dist-test-slave-7lbf"
server_key: "c3ea77227f1f66ed2f4da52cb97e6d3c"
server_key_iv: "605bcfc036687d9d90528d4ea2b90330"
server_key_version: "encryptionkey@0"
I20260810 19:19:22.942943 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:22.954180 10816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:22.958246 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:22.958503 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:22.961030 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:22.961136 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:22.961277 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:22.961364 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:19:22 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389562, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:22.983954 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.961413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:59367 (local address 127.7.43.62:38253)
0810 19:19:22.961635 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:22.961641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:22.963053 (+ 1412us) server_negotiation.cc:407] Connection header received
0810 19:19:22.965162 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:22.965168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:22.965256 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:22.965378 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:22.967689 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.968873 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.971634 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.972273 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.972415 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:22.977439 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:22.977447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:22.977452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:22.977548 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:22.980772 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:22.981556 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:22.981562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:22.981565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:22.981741 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:22.982224 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:22.982230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:22.982233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:22.982517 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:22.982656 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:22.983357 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 19:19:22.983589 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:19:22.985237 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.962038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.7.43.3:59367)
0810 19:19:22.962867 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:22.962919 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:22.964857 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:22.965655 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:22.965667 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:22.966448 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:22.967473 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.967501 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.969063 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.969068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.971433 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.971453 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.972528 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.972536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.972849 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:22.973684 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:22.973722 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:22.977130 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:22.980905 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:22.980914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:22.980938 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:22.981390 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:22.981890 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:22.981894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:22.981898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:22.982100 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:22.982644 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:22.982652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:22.982983 (+ 331us) client_negotiation.cc:769] Sending connection context
0810 19:19:22.983208 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 19:19:22.983726 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":344,"threads_started":1}
I20260810 19:19:22.986984 10590 heartbeater.cc:503] Master 127.7.43.62:38253 was elected leader, sending a full tablet report...
I20260810 19:19:23.003755 10816 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:36587
I20260810 19:19:23.003809 11042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:36587 every 8 connection(s)
I20260810 19:19:23.005033 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:23.007335 11043 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38253
I20260810 19:19:23.007592 11043 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:23.008456 11043 heartbeater.cc:511] Master 127.7.43.62:38253 requested a full tablet report, sending...
I20260810 19:19:23.008440 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10816
I20260810 19:19:23.008641 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:23.009181 7340 external_mini_cluster.cc:1428] Setting key e9c05d0855354cc705678f0693544716
I20260810 19:19:23.009799 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.991898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.7.43.3:59521)
0810 19:19:22.992603 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:22.992624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:22.992757 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:22.993072 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:22.993076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:22.993735 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:22.994209 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.994224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.995439 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.995445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.996687 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:22.996703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.997746 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.997754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:22.998002 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.002495 (+ 4493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:23.002514 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:23.003511 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:23.007010 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:23.007015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:23.007018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:23.007517 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:23.007940 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:23.007944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:23.007946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:23.008076 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:23.008682 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:23.008687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:23.008871 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.008992 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.009322 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":395,"threads_started":1}
I20260810 19:19:23.010102 10245 ts_manager.cc:194] Registered new tserver with Master: 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:23.010253 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:22.991750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:59521 (local address 127.7.43.62:38253)
0810 19:19:22.991913 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:22.991918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:22.992706 (+ 788us) server_negotiation.cc:407] Connection header received
0810 19:19:22.992875 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:22.992880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:22.992947 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:22.993024 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:22.994346 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.995304 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.996830 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:22.997345 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:22.997479 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.003705 (+ 6226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:23.003710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:23.003713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:23.003745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:23.006664 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:23.007657 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:23.007663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:23.007666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:23.007829 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:23.008185 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:23.008190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:23.008194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:23.008491 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:23.008629 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.009120 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.009341 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:19:23.011243 10245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:59367
I20260810 19:19:23.012648 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:19:23 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:19:23 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:23.065873 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.052361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.7.43.62:38253)
0810 19:19:23.052532 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.052537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.052923 (+ 386us) server_negotiation.cc:407] Connection header received
0810 19:19:23.053104 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.053109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.053183 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.053273 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:23.054705 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.055435 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.056746 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.057221 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.057318 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.061090 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:23.061095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:23.061098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:23.061136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:23.063631 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:23.064235 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:23.064239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:23.064241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:23.064357 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:23.064629 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:23.064632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:23.064634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:23.064889 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:23.065089 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.065293 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.065484 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:19:23.070006 10244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:23.076823 10244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:23.102231 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.090111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.7.43.1:34341)
0810 19:19:23.092558 (+ 2447us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.092567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.092597 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:19:23.092717 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.092725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.092990 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.093212 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:23.095231 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.096680 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.099487 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.100740 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.100872 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.101091 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.101385 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.101953 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2289,"thread_start_us":2129,"threads_started":1}
I20260810 19:19:23.102612 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.090484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.7.43.2:34427)
0810 19:19:23.092102 (+ 1618us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.092113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.094300 (+ 2187us) server_negotiation.cc:407] Connection header received
0810 19:19:23.094618 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.094626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.094899 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.095081 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:23.095747 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.097096 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.099725 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.100771 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.100898 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.101105 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.102175 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.102345 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438,"thread_start_us":1306,"threads_started":1}
I20260810 19:19:23.102985 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.089759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:34427 (local address 127.0.0.1:33726)
0810 19:19:23.094164 (+ 4405us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:23.094189 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:23.094424 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:23.095181 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:23.095185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:23.095209 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:23.095574 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.095587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.097257 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.097262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.099585 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.099596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.101257 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.101263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.101421 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.101456 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.101615 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.101745 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4262,"mutex_wait_us":58,"thread_start_us":319,"threads_started":1}
I20260810 19:19:23.102946 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.089701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34341 (local address 127.0.0.1:59538)
0810 19:19:23.090580 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:23.090632 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:23.090819 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:23.093246 (+ 2427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:23.093254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:23.093290 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:23.095004 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.095019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.096801 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.096806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.099317 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.099332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.100926 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.100933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.101124 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.101169 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.101290 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.101475 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":38,"thread_start_us":3891,"threads_started":1}
I20260810 19:19:23.103525 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.091990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.7.43.3:36587)
0810 19:19:23.094567 (+ 2577us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.094575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.095422 (+ 847us) server_negotiation.cc:407] Connection header received
0810 19:19:23.095506 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.095511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.095659 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.095869 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:23.096457 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.098005 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.100365 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.101780 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.101898 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.102107 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.103054 (+ 947us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.103208 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2407,"thread_start_us":2203,"threads_started":1}
I20260810 19:19:23.104285 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.091374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.0.0.1:33608)
0810 19:19:23.095189 (+ 3815us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:23.095206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:23.095321 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:23.095843 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:23.095848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:23.095873 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:23.096252 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.096263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.098187 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.098192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.100190 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.100206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.102249 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.102255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.102435 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.102468 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.102554 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.102667 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3697,"mutex_wait_us":50,"thread_start_us":3556,"threads_started":1}
I20260810 19:19:23.106383 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 48e94ff7b36845c2858a81c4909760c9 (DEFAULT_TABLE table=test-table [id=cd1b9bccd27c4a0985fd8703653a3456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:23.106386 10701 tablet_service.cc:1467] Processing CreateTablet for tablet 48e94ff7b36845c2858a81c4909760c9 (DEFAULT_TABLE table=test-table [id=cd1b9bccd27c4a0985fd8703653a3456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:23.107672 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e94ff7b36845c2858a81c4909760c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:23.107717 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e94ff7b36845c2858a81c4909760c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:23.110489 10920 tablet_service.cc:1467] Processing CreateTablet for tablet 48e94ff7b36845c2858a81c4909760c9 (DEFAULT_TABLE table=test-table [id=cd1b9bccd27c4a0985fd8703653a3456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:23.111975 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e94ff7b36845c2858a81c4909760c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:23.119777 11074 tablet_bootstrap.cc:492] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Bootstrap starting.
I20260810 19:19:23.122179 11076 tablet_bootstrap.cc:492] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Bootstrap starting.
I20260810 19:19:23.123848 11074 tablet_bootstrap.cc:654] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:23.125196 11074 log.cc:824] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:23.126120 11076 tablet_bootstrap.cc:654] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:23.126528 11079 tablet_bootstrap.cc:492] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Bootstrap starting.
I20260810 19:19:23.127486 11076 log.cc:824] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:23.129189 11074 tablet_bootstrap.cc:492] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: No bootstrap required, opened a new log
I20260810 19:19:23.129487 11074 ts_tablet_manager.cc:1397] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260810 19:19:23.130357 11079 tablet_bootstrap.cc:654] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:23.130761 11076 tablet_bootstrap.cc:492] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: No bootstrap required, opened a new log
I20260810 19:19:23.131062 11076 ts_tablet_manager.cc:1397] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 19:19:23.131389 11079 log.cc:824] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:23.139930 11079 tablet_bootstrap.cc:492] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: No bootstrap required, opened a new log
I20260810 19:19:23.140202 11079 ts_tablet_manager.cc:1397] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260810 19:19:23.140091 11074 raft_consensus.cc:348] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.140583 11074 raft_consensus.cc:374] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:23.140683 11074 raft_consensus.cc:729] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d2627090cd49788c1bbacd8dce32a3, State: Initialized, Role: FOLLOWER
I20260810 19:19:23.141376 11074 consensus_queue.cc:260] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.141335 11076 raft_consensus.cc:348] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.141780 11076 raft_consensus.cc:374] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:23.141904 11076 raft_consensus.cc:729] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Initialized, Role: FOLLOWER
I20260810 19:19:23.142776 11076 consensus_queue.cc:260] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.143313 10809 heartbeater.cc:503] Master 127.7.43.62:38253 was elected leader, sending a full tablet report...
I20260810 19:19:23.144033 11074 ts_tablet_manager.cc:1428] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 19:19:23.145752 11076 ts_tablet_manager.cc:1428] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260810 19:19:23.150751 11079 raft_consensus.cc:348] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.151098 11079 raft_consensus.cc:374] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:23.151167 11079 raft_consensus.cc:729] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Initialized, Role: FOLLOWER
I20260810 19:19:23.151672 11079 consensus_queue.cc:260] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.152693 11043 heartbeater.cc:503] Master 127.7.43.62:38253 was elected leader, sending a full tablet report...
I20260810 19:19:23.153239 11079 ts_tablet_manager.cc:1428] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260810 19:19:23.229652 10591 tablet.cc:2369] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:19:23.251611 10810 tablet.cc:2369] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:19:23.256740 11044 tablet.cc:2369] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:23.441344 11094 raft_consensus.cc:482] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:23.441694 11094 raft_consensus.cc:504] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.443311 11094 leader_election.cc:288] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341), 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427)
I20260810 19:19:23.448808 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.443624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:34427 (local address 127.7.43.3:51649)
0810 19:19:23.443913 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:23.443933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:23.444060 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:23.444399 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:23.444403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:23.444430 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:23.444629 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.444638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.445638 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.445643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.447015 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.447028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.448093 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.448100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.448265 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.448289 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.448366 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.448501 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"spinlock_wait_cycles":896}
I20260810 19:19:23.448823 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.443811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:51649 (local address 127.7.43.2:34427)
0810 19:19:23.444053 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.444059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.444079 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:19:23.444210 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.444214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.444272 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.444354 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:23.444763 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.445494 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.447165 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.447932 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.448061 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.448160 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.448408 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.448527 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 19:19:23.449411 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.443766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:50011 (local address 127.7.43.1:34341)
0810 19:19:23.443965 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.443972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.444472 (+ 500us) server_negotiation.cc:407] Connection header received
0810 19:19:23.444672 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.444676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.444748 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.444822 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:23.445328 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.446072 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.447796 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.448552 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.448686 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.448779 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.449119 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.449246 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:19:23.449841 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e94ff7b36845c2858a81c4909760c9" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3" is_pre_election: true
I20260810 19:19:23.450037 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.443730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34341 (local address 127.7.43.3:50011)
0810 19:19:23.444362 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:23.444375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:23.444536 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:23.444855 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:23.444859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:23.444881 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:23.445179 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.445188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.446198 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.446202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.447655 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.447668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.448769 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.448775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.448947 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.448981 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.449600 (+ 619us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.449743 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"spinlock_wait_cycles":896,"thread_start_us":355,"threads_started":1}
I20260810 19:19:23.450345 10734 raft_consensus.cc:2454] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 0.
I20260810 19:19:23.451021 10841 leader_election.cc:302] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 4a41b69697184a378c82e6ae3cb16a03; no voters:
I20260810 19:19:23.450973 10520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e94ff7b36845c2858a81c4909760c9" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5899515cff904a8397fa4260be94ebc9" is_pre_election: true
I20260810 19:19:23.451403 11094 raft_consensus.cc:2781] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:23.451442 10520 raft_consensus.cc:2454] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 0.
I20260810 19:19:23.451567 11094 raft_consensus.cc:482] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:23.451671 11094 raft_consensus.cc:3037] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:23.454005 11094 raft_consensus.cc:504] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.454566 11094 leader_election.cc:288] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 election: Requested vote from peers 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341), 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427)
I20260810 19:19:23.454957 10520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e94ff7b36845c2858a81c4909760c9" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5899515cff904a8397fa4260be94ebc9"
I20260810 19:19:23.455044 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e94ff7b36845c2858a81c4909760c9" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3"
I20260810 19:19:23.455159 10520 raft_consensus.cc:3037] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:23.455229 10734 raft_consensus.cc:3037] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:23.458321 10734 raft_consensus.cc:2454] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 1.
I20260810 19:19:23.458740 10841 leader_election.cc:302] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 4a41b69697184a378c82e6ae3cb16a03; no voters:
I20260810 19:19:23.459025 11094 raft_consensus.cc:2781] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:23.459246 10520 raft_consensus.cc:2454] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 1.
I20260810 19:19:23.459543 11094 raft_consensus.cc:686] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 LEADER]: Becoming Leader. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Running, Role: LEADER
I20260810 19:19:23.460057 11094 consensus_queue.cc:237] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.464496 10244 catalog_manager.cc:5495] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } health_report { overall_health: HEALTHY } } }
Aug 10 19:19:23 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389561, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:23.584546 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.571705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38253 (local address 127.0.0.1:46742)
0810 19:19:23.572385 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:23.572401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:23.572533 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:23.572811 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:23.572814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:23.573531 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:23.573779 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.573788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.574835 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.574838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.575786 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:23.575797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.576438 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.576443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:23.576571 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.577185 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:23.577207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:23.579372 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:23.582392 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:23.582403 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:23.582409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:23.582829 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:23.583137 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:23.583140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:23.583143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:23.583287 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:23.583746 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:23.583752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:23.583897 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 19:19:23.584026 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 19:19:23.584219 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":348,"threads_started":1}
I20260810 19:19:23.584662 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:23.571708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.7.43.62:38253)
0810 19:19:23.572290 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:23.572295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:23.572477 (+ 182us) server_negotiation.cc:407] Connection header received
0810 19:19:23.572642 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:23.572646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:23.572703 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:23.572782 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:23.573926 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.574721 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.575897 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:23.576283 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:23.576367 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:23.579572 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:23.579577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:23.579580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:23.579608 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:23.582284 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:23.582945 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:23.582949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:23.582951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:23.583062 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:23.583376 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:23.583379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:23.583381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:23.583641 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:23.583747 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:23.584106 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 19:19:23.584345 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":469,"thread_start_us":321,"threads_started":1}
I20260810 19:19:23.589402 10244 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:46742:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 19:19:23.590497 10244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:23.597051 10701 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa820a9c73d4cbba8bbe7d1a8b6e470 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0ccdf07e1a1411c9b05be43574890f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:19:23.597051 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa820a9c73d4cbba8bbe7d1a8b6e470 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0ccdf07e1a1411c9b05be43574890f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:19:23.597337 10920 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa820a9c73d4cbba8bbe7d1a8b6e470 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0ccdf07e1a1411c9b05be43574890f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:19:23.598073 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa820a9c73d4cbba8bbe7d1a8b6e470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:23.598073 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa820a9c73d4cbba8bbe7d1a8b6e470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:23.598251 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa820a9c73d4cbba8bbe7d1a8b6e470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:23.604130 11079 tablet_bootstrap.cc:492] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: Bootstrap starting.
I20260810 19:19:23.605409 11074 tablet_bootstrap.cc:492] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: Bootstrap starting.
I20260810 19:19:23.605681 11079 tablet_bootstrap.cc:654] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:23.605756 11076 tablet_bootstrap.cc:492] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: Bootstrap starting.
I20260810 19:19:23.607313 11076 tablet_bootstrap.cc:654] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:23.607705 11079 tablet_bootstrap.cc:492] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: No bootstrap required, opened a new log
I20260810 19:19:23.607846 11079 ts_tablet_manager.cc:1397] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260810 19:19:23.608596 11079 raft_consensus.cc:348] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.608852 11079 raft_consensus.cc:374] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:23.608950 11079 raft_consensus.cc:729] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Initialized, Role: FOLLOWER
I20260810 19:19:23.609174 11079 consensus_queue.cc:260] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.609722 11074 tablet_bootstrap.cc:654] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:23.609644 11094 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:23.609813 11079 ts_tablet_manager.cc:1428] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 19:19:23.609898 11094 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:23.611641 11076 tablet_bootstrap.cc:492] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: No bootstrap required, opened a new log
I20260810 19:19:23.611779 11076 ts_tablet_manager.cc:1397] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260810 19:19:23.612490 11076 raft_consensus.cc:348] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.612697 11076 raft_consensus.cc:374] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:23.612790 11076 raft_consensus.cc:729] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Initialized, Role: FOLLOWER
I20260810 19:19:23.613000 11076 consensus_queue.cc:260] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.613598 11076 ts_tablet_manager.cc:1428] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:19:23.613420 11091 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:23.613826 11091 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:23.621287 11074 tablet_bootstrap.cc:492] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: No bootstrap required, opened a new log
I20260810 19:19:23.621411 11074 ts_tablet_manager.cc:1397] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260810 19:19:23.622072 11074 raft_consensus.cc:348] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.622262 11074 raft_consensus.cc:374] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:23.622326 11074 raft_consensus.cc:729] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d2627090cd49788c1bbacd8dce32a3, State: Initialized, Role: FOLLOWER
I20260810 19:19:23.622529 11074 consensus_queue.cc:260] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.623063 11074 ts_tablet_manager.cc:1428] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 19:19:23.622961 11087 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:23.623346 11087 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:23.954038 11094 consensus_queue.cc:1035] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:23.961761 11094 consensus_queue.cc:1035] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:23.975312 11125 raft_consensus.cc:482] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:23.975512 11125 raft_consensus.cc:504] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.976249 11125 leader_election.cc:288] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341), 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427)
I20260810 19:19:23.976706 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa820a9c73d4cbba8bbe7d1a8b6e470" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3" is_pre_election: true
I20260810 19:19:23.976732 10520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa820a9c73d4cbba8bbe7d1a8b6e470" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5899515cff904a8397fa4260be94ebc9" is_pre_election: true
I20260810 19:19:23.976969 10734 raft_consensus.cc:2454] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 0.
I20260810 19:19:23.976979 10520 raft_consensus.cc:2454] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 0.
I20260810 19:19:23.977375 10841 leader_election.cc:302] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 4a41b69697184a378c82e6ae3cb16a03; no voters:
I20260810 19:19:23.977697 11125 raft_consensus.cc:2781] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:23.977806 11125 raft_consensus.cc:482] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:23.977891 11125 raft_consensus.cc:3037] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:23.980118 11125 raft_consensus.cc:504] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.980686 11125 leader_election.cc:288] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 election: Requested vote from peers 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341), 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427)
I20260810 19:19:23.981218 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa820a9c73d4cbba8bbe7d1a8b6e470" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3"
I20260810 19:19:23.981205 10520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa820a9c73d4cbba8bbe7d1a8b6e470" candidate_uuid: "4a41b69697184a378c82e6ae3cb16a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5899515cff904a8397fa4260be94ebc9"
I20260810 19:19:23.981467 10520 raft_consensus.cc:3037] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:23.981465 10734 raft_consensus.cc:3037] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:23.983505 10734 raft_consensus.cc:2454] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 1.
I20260810 19:19:23.983881 10841 leader_election.cc:302] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 4a41b69697184a378c82e6ae3cb16a03; no voters:
I20260810 19:19:23.984148 11125 raft_consensus.cc:2781] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:23.984282 11125 raft_consensus.cc:686] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 LEADER]: Becoming Leader. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Running, Role: LEADER
I20260810 19:19:23.984678 10520 raft_consensus.cc:2454] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a41b69697184a378c82e6ae3cb16a03 in term 1.
I20260810 19:19:23.984596 11125 consensus_queue.cc:237] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:23.986228 11097 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: TxnStatusTablet state changed. Reason: New leader 4a41b69697184a378c82e6ae3cb16a03. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:23.986522 11097 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:19:23.987115 11132 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 19:19:23.987488 11132 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 19:19:23.987831 10244 catalog_manager.cc:5495] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } health_report { overall_health: HEALTHY } } }
I20260810 19:19:24.067478 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.061615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.0.0.1:33624)
0810 19:19:24.061844 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:24.061862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:24.062007 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:24.062708 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:24.062712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:24.062725 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:24.062943 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.062954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.064013 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.064019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.065016 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.065027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.065761 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.065767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.065930 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.065990 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.066914 (+ 924us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.066937 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:19:24.067015 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 19:19:24.067177 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 19:19:24.067526 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.061706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.7.43.3:36587)
0810 19:19:24.062340 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.062344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.062362 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:19:24.062448 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.062451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.062524 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.062667 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:24.063096 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.063880 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.065149 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.065612 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.065724 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.066184 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.066782 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.066931 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.067055 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.067131 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":324,"threads_started":1}
I20260810 19:19:24.075316 10520 raft_consensus.cc:1261] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Refusing update from remote peer 4a41b69697184a378c82e6ae3cb16a03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:24.075338 10734 raft_consensus.cc:1261] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Refusing update from remote peer 4a41b69697184a378c82e6ae3cb16a03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:24.076009 11097 consensus_queue.cc:1035] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:24.076342 11125 consensus_queue.cc:1035] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:24.081867 11087 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: TxnStatusTablet state changed. Reason: New leader 4a41b69697184a378c82e6ae3cb16a03. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:24.082214 11087 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:24.086984 11122 mvcc.cc:205] Tried to move back new op lower bound from 7317051654444601344 to 7317051654083424256. Current Snapshot: MvccSnapshot[applied={T|T < 7317051654444601344}]
I20260810 19:19:24.095860 11087 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:24.096174 11087 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 07d2627090cd49788c1bbacd8dce32a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:24.098626 11091 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: TxnStatusTablet state changed. Reason: New leader 4a41b69697184a378c82e6ae3cb16a03. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:24.098933 11091 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:24.100804 11097 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:24.101083 11097 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:19:24.101949 11125 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:24.102238 11125 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 4a41b69697184a378c82e6ae3cb16a03: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:19:24.104346 11091 tablet_replica.cc:439] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a41b69697184a378c82e6ae3cb16a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } } }
I20260810 19:19:24.104600 11091 tablet_replica.cc:442] T 7aa820a9c73d4cbba8bbe7d1a8b6e470 P 5899515cff904a8397fa4260be94ebc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:19:24.126884 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.117275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.7.43.3:36587)
0810 19:19:24.117506 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.117536 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.121662 (+ 4126us) server_negotiation.cc:407] Connection header received
0810 19:19:24.121908 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.121914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.122009 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.122110 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:24.122566 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.123486 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.125012 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.125534 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.125676 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.126105 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.126220 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.126324 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.126478 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.126563 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:19:24.143648 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.138340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:38821 (local address 127.7.43.3:36587)
0810 19:19:24.138506 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.138512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.138912 (+ 400us) server_negotiation.cc:407] Connection header received
0810 19:19:24.139091 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.139094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.139161 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.139232 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:24.139707 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.140499 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.141983 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.142406 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.142515 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.142919 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.143022 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.143103 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.143224 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.143304 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260810 19:19:24.143709 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.138295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.7.43.3:38821)
0810 19:19:24.138816 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:24.138831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:24.138975 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:24.139257 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:24.139260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:24.139271 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:24.139557 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.139567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.140646 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.140649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.141850 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.141863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.142547 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.142553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.142689 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.142745 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.143115 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.143126 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 19:19:24.143194 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 19:19:24.143327 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260810 19:19:24.212476 10245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:24.212728 10245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:24.216262 10245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Sending DeleteTablet for 3 replicas of tablet 48e94ff7b36845c2858a81c4909760c9
I20260810 19:19:24.217388 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e94ff7b36845c2858a81c4909760c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59538
I20260810 19:19:24.217543 10920 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e94ff7b36845c2858a81c4909760c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33608
I20260810 19:19:24.217540 10701 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e94ff7b36845c2858a81c4909760c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33726
I20260810 19:19:24.223637 11167 tablet_replica.cc:330] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: stopping tablet replica
I20260810 19:19:24.223814 11166 tablet_replica.cc:330] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: stopping tablet replica
I20260810 19:19:24.224151 11167 raft_consensus.cc:2229] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:24.224184 11166 raft_consensus.cc:2229] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:24.224531 11166 raft_consensus.cc:2258] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:24.224542 11167 raft_consensus.cc:2258] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:24.224571 10245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:24.225360 10245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:24.226715 11166 ts_tablet_manager.cc:1905] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:24.226850 11168 tablet_replica.cc:330] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: stopping tablet replica
I20260810 19:19:24.229552 11167 ts_tablet_manager.cc:1905] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:24.233954 11166 ts_tablet_manager.cc:1918] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:24.234124 11166 log.cc:1196] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/48e94ff7b36845c2858a81c4909760c9
I20260810 19:19:24.234658 11166 ts_tablet_manager.cc:1939] T 48e94ff7b36845c2858a81c4909760c9 P 5899515cff904a8397fa4260be94ebc9: Deleting consensus metadata
I20260810 19:19:24.235622 10701 tablet_service.cc:1467] Processing CreateTablet for tablet 460f7eb032e243dd94873173604dce8b (DEFAULT_TABLE table=test-table [id=b72aff2b3f224e4fbd29961f07834179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:24.236119 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460f7eb032e243dd94873173604dce8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:24.236505 10224 catalog_manager.cc:4841] TS 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341): tablet 48e94ff7b36845c2858a81c4909760c9 (table test-table [id=cd1b9bccd27c4a0985fd8703653a3456]) successfully deleted
I20260810 19:19:24.236783 11167 ts_tablet_manager.cc:1918] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:24.227252 11168 raft_consensus.cc:2229] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:24.236965 11167 log.cc:1196] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/48e94ff7b36845c2858a81c4909760c9
I20260810 19:19:24.237659 11167 ts_tablet_manager.cc:1939] T 48e94ff7b36845c2858a81c4909760c9 P 07d2627090cd49788c1bbacd8dce32a3: Deleting consensus metadata
I20260810 19:19:24.237753 11168 raft_consensus.cc:2258] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:24.239782 11168 ts_tablet_manager.cc:1905] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:24.239874 10220 catalog_manager.cc:4841] TS 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427): tablet 48e94ff7b36845c2858a81c4909760c9 (table test-table [id=cd1b9bccd27c4a0985fd8703653a3456]) successfully deleted
I20260810 19:19:24.240653 10920 tablet_service.cc:1467] Processing CreateTablet for tablet 460f7eb032e243dd94873173604dce8b (DEFAULT_TABLE table=test-table [id=b72aff2b3f224e4fbd29961f07834179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:24.241176 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 460f7eb032e243dd94873173604dce8b (DEFAULT_TABLE table=test-table [id=b72aff2b3f224e4fbd29961f07834179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:24.241761 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460f7eb032e243dd94873173604dce8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:24.242115 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460f7eb032e243dd94873173604dce8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:24.251147 11170 tablet_bootstrap.cc:492] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Bootstrap starting.
I20260810 19:19:24.254990 11171 tablet_bootstrap.cc:492] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Bootstrap starting.
I20260810 19:19:24.255052 11168 ts_tablet_manager.cc:1918] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:24.255245 11168 log.cc:1196] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/48e94ff7b36845c2858a81c4909760c9
I20260810 19:19:24.255867 11168 ts_tablet_manager.cc:1939] T 48e94ff7b36845c2858a81c4909760c9 P 4a41b69697184a378c82e6ae3cb16a03: Deleting consensus metadata
I20260810 19:19:24.256471 11170 tablet_bootstrap.cc:654] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:24.257871 11171 tablet_bootstrap.cc:654] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:24.257867 10220 catalog_manager.cc:4841] TS 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587): tablet 48e94ff7b36845c2858a81c4909760c9 (table test-table [id=cd1b9bccd27c4a0985fd8703653a3456]) successfully deleted
I20260810 19:19:24.262805 11172 tablet_bootstrap.cc:492] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Bootstrap starting.
I20260810 19:19:24.264973 11172 tablet_bootstrap.cc:654] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:24.266590 11171 tablet_bootstrap.cc:492] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: No bootstrap required, opened a new log
I20260810 19:19:24.266822 11171 ts_tablet_manager.cc:1397] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260810 19:19:24.267731 11171 raft_consensus.cc:348] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.268172 11171 raft_consensus.cc:374] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:24.268325 11171 raft_consensus.cc:729] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Initialized, Role: FOLLOWER
I20260810 19:19:24.268716 11171 consensus_queue.cc:260] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.270496 11171 ts_tablet_manager.cc:1428] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 19:19:24.272266 11172 tablet_bootstrap.cc:492] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: No bootstrap required, opened a new log
I20260810 19:19:24.272333 11170 tablet_bootstrap.cc:492] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: No bootstrap required, opened a new log
I20260810 19:19:24.272488 11170 ts_tablet_manager.cc:1397] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.006s
I20260810 19:19:24.272590 11172 ts_tablet_manager.cc:1397] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260810 19:19:24.273267 11170 raft_consensus.cc:348] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.273548 11170 raft_consensus.cc:374] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:24.273649 11170 raft_consensus.cc:729] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d2627090cd49788c1bbacd8dce32a3, State: Initialized, Role: FOLLOWER
I20260810 19:19:24.273420 11172 raft_consensus.cc:348] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.273953 11172 raft_consensus.cc:374] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:24.274096 11172 raft_consensus.cc:729] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Initialized, Role: FOLLOWER
I20260810 19:19:24.273953 11170 consensus_queue.cc:260] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.274533 11170 ts_tablet_manager.cc:1428] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:19:24.275058 11172 consensus_queue.cc:260] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.275772 11172 ts_tablet_manager.cc:1428] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 19:19:24.361605 11087 raft_consensus.cc:482] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:24.361863 11087 raft_consensus.cc:504] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.363458 11087 leader_election.cc:288] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341), 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:24.381247 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.364155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:53141 (local address 127.7.43.3:36587)
0810 19:19:24.364348 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.364354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.374639 (+ 10285us) server_negotiation.cc:407] Connection header received
0810 19:19:24.374885 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.374891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.374968 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.375024 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:24.375756 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.376597 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.378737 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.380089 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.380210 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.380330 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.380837 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.380976 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:19:24.381628 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.363911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:45539 (local address 127.7.43.1:34341)
0810 19:19:24.375202 (+ 11291us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.375206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.375225 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:19:24.375282 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.375285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.375339 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.375423 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:24.376121 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.377307 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.379534 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.380647 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.380761 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.380885 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.381149 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.381279 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11181,"thread_start_us":11033,"threads_started":1}
I20260810 19:19:24.381683 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.363680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34341 (local address 127.7.43.2:45539)
0810 19:19:24.374289 (+ 10609us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:24.374318 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:24.374525 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:24.375478 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:24.375482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:24.375517 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:24.375923 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.375935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.377458 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.377464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.379325 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.379339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.380829 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.380835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.381001 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.381024 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:19:24.381103 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:19:24.381229 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10437,"thread_start_us":10941,"threads_started":1}
I20260810 19:19:24.382238 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.363767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.7.43.2:53141)
0810 19:19:24.374526 (+ 10759us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:24.374544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:24.374744 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:24.375201 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:24.375206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:24.375229 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:24.375566 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.375581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.376733 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.376738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.378569 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.378584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.380472 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.380479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.380646 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.380681 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:19:24.381690 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0810 19:19:24.381834 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10633,"thread_start_us":10150,"threads_started":1}
I20260810 19:19:24.382483 10519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460f7eb032e243dd94873173604dce8b" candidate_uuid: "07d2627090cd49788c1bbacd8dce32a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5899515cff904a8397fa4260be94ebc9" is_pre_election: true
I20260810 19:19:24.382807 10519 raft_consensus.cc:2454] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d2627090cd49788c1bbacd8dce32a3 in term 0.
I20260810 19:19:24.383015 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460f7eb032e243dd94873173604dce8b" candidate_uuid: "07d2627090cd49788c1bbacd8dce32a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a41b69697184a378c82e6ae3cb16a03" is_pre_election: true
I20260810 19:19:24.383258 10621 leader_election.cc:302] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 5899515cff904a8397fa4260be94ebc9; no voters:
I20260810 19:19:24.383421 10960 raft_consensus.cc:2454] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d2627090cd49788c1bbacd8dce32a3 in term 0.
I20260810 19:19:24.383658 11087 raft_consensus.cc:2781] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:24.383769 11087 raft_consensus.cc:482] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:24.383878 11087 raft_consensus.cc:3037] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:24.386082 11087 raft_consensus.cc:504] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.386654 11087 leader_election.cc:288] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [CANDIDATE]: Term 1 election: Requested vote from peers 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341), 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:24.387149 10519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460f7eb032e243dd94873173604dce8b" candidate_uuid: "07d2627090cd49788c1bbacd8dce32a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5899515cff904a8397fa4260be94ebc9"
I20260810 19:19:24.387197 10960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460f7eb032e243dd94873173604dce8b" candidate_uuid: "07d2627090cd49788c1bbacd8dce32a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a41b69697184a378c82e6ae3cb16a03"
I20260810 19:19:24.387336 10519 raft_consensus.cc:3037] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:24.387434 10960 raft_consensus.cc:3037] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:24.389602 10519 raft_consensus.cc:2454] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d2627090cd49788c1bbacd8dce32a3 in term 1.
I20260810 19:19:24.390058 10621 leader_election.cc:302] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 5899515cff904a8397fa4260be94ebc9; no voters:
I20260810 19:19:24.390311 11087 raft_consensus.cc:2781] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:24.390793 11087 raft_consensus.cc:686] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 1 LEADER]: Becoming Leader. State: Replica: 07d2627090cd49788c1bbacd8dce32a3, State: Running, Role: LEADER
I20260810 19:19:24.390807 10960 raft_consensus.cc:2454] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d2627090cd49788c1bbacd8dce32a3 in term 1.
I20260810 19:19:24.391278 11087 consensus_queue.cc:237] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:24.394829 10244 catalog_manager.cc:5495] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "07d2627090cd49788c1bbacd8dce32a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } health_report { overall_health: UNKNOWN } } }
I20260810 19:19:24.428608 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.422822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.7.43.2:34427)
0810 19:19:24.423579 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.423583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.423601 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:19:24.423676 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.423679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.423754 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.423902 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:24.424339 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.425115 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.426434 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.426860 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.426970 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.427399 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.427978 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.428118 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.428206 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.428284 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":479,"threads_started":1}
I20260810 19:19:24.444718 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.438498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:50821 (local address 127.7.43.3:36587)
0810 19:19:24.438686 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.438693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.439218 (+ 525us) server_negotiation.cc:407] Connection header received
0810 19:19:24.439409 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.439413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.439512 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.439583 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:24.440166 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.440936 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.442943 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.443417 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.443529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.444038 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.444170 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.444238 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.444381 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.444468 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:19:24.444900 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.438326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.7.43.2:50821)
0810 19:19:24.439111 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:24.439132 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:24.439281 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:24.439621 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:24.439625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:24.439638 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:24.439998 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.440010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.441087 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.441093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.442806 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.442822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.443609 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.443616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.443806 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.443878 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.444266 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.444277 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 19:19:24.444362 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 19:19:24.444511 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":451,"threads_started":1}
I20260810 19:19:24.454322 10520 raft_consensus.cc:1261] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Refusing update from remote peer 07d2627090cd49788c1bbacd8dce32a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:24.454488 10960 raft_consensus.cc:1261] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Refusing update from remote peer 07d2627090cd49788c1bbacd8dce32a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:24.454916 11087 consensus_queue.cc:1035] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:24.455260 11184 consensus_queue.cc:1035] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:24.461496 11127 mvcc.cc:205] Tried to move back new op lower bound from 7317051655998533632 to 7317051655750627328. Current Snapshot: MvccSnapshot[applied={T|T < 7317051655998533632}]
I20260810 19:19:24.462342 11129 mvcc.cc:205] Tried to move back new op lower bound from 7317051655998533632 to 7317051655750627328. Current Snapshot: MvccSnapshot[applied={T|T < 7317051655998533632}]
I20260810 19:19:24.484913 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.479266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:34427 (local address 127.7.43.3:48209)
0810 19:19:24.479440 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:24.479468 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:24.479605 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:24.480090 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:24.480096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:24.480112 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:24.480290 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.480303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.481748 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.481752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.483115 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:24.483129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.483885 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.483890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:24.484045 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.484084 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.484459 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.484472 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 19:19:24.484553 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:19:24.484677 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 19:19:24.485412 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:24.479582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:48209 (local address 127.7.43.2:34427)
0810 19:19:24.479840 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:24.479847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:24.479863 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:19:24.479916 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:24.479919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:24.479977 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:24.480050 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:24.480437 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.481616 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.483234 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:24.483756 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:24.483862 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:24.484224 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:24.484352 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:24.484450 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:24.485123 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 19:19:24.485198 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
W20260810 19:19:24.486356 11129 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 19:19:26.093547 10245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:26.093792 10245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:26.098728 10245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Sending DeleteTablet for 3 replicas of tablet 460f7eb032e243dd94873173604dce8b
I20260810 19:19:26.099555 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 460f7eb032e243dd94873173604dce8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:26 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59538
I20260810 19:19:26.099684 10701 tablet_service.cc:1514] Processing DeleteTablet for tablet 460f7eb032e243dd94873173604dce8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:26 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33726
I20260810 19:19:26.099663 10920 tablet_service.cc:1514] Processing DeleteTablet for tablet 460f7eb032e243dd94873173604dce8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:26 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33608
I20260810 19:19:26.111045 11234 tablet_replica.cc:330] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: stopping tablet replica
I20260810 19:19:26.111327 11234 raft_consensus.cc:2229] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:26.112023 11234 raft_consensus.cc:2258] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:26.112074 11232 tablet_replica.cc:330] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: stopping tablet replica
I20260810 19:19:26.112594 11232 raft_consensus.cc:2229] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:26.113030 11232 raft_consensus.cc:2258] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:26.113281 11234 ts_tablet_manager.cc:1905] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:26.114347 11232 ts_tablet_manager.cc:1905] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:26.121397 11232 ts_tablet_manager.cc:1918] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:19:26.121748 11232 log.cc:1196] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/460f7eb032e243dd94873173604dce8b
I20260810 19:19:26.121982 11234 ts_tablet_manager.cc:1918] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:19:26.122216 11234 log.cc:1196] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/460f7eb032e243dd94873173604dce8b
I20260810 19:19:26.122359 11233 tablet_replica.cc:330] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: stopping tablet replica
I20260810 19:19:26.122605 11232 ts_tablet_manager.cc:1939] T 460f7eb032e243dd94873173604dce8b P 5899515cff904a8397fa4260be94ebc9: Deleting consensus metadata
I20260810 19:19:26.122702 11233 raft_consensus.cc:2229] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:26.122947 11234 ts_tablet_manager.cc:1939] T 460f7eb032e243dd94873173604dce8b P 07d2627090cd49788c1bbacd8dce32a3: Deleting consensus metadata
I20260810 19:19:26.123175 11233 raft_consensus.cc:2258] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:26.124079 10220 catalog_manager.cc:4841] TS 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427): tablet 460f7eb032e243dd94873173604dce8b (table test-table [id=b72aff2b3f224e4fbd29961f07834179]) successfully deleted
I20260810 19:19:26.124079 10224 catalog_manager.cc:4841] TS 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341): tablet 460f7eb032e243dd94873173604dce8b (table test-table [id=b72aff2b3f224e4fbd29961f07834179]) successfully deleted
I20260810 19:19:26.126179 11233 ts_tablet_manager.cc:1905] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:26.131942 11233 ts_tablet_manager.cc:1918] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:19:26.132148 11233 log.cc:1196] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/460f7eb032e243dd94873173604dce8b
I20260810 19:19:26.132851 11233 ts_tablet_manager.cc:1939] T 460f7eb032e243dd94873173604dce8b P 4a41b69697184a378c82e6ae3cb16a03: Deleting consensus metadata
I20260810 19:19:26.134030 10220 catalog_manager.cc:4841] TS 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587): tablet 460f7eb032e243dd94873173604dce8b (table test-table [id=b72aff2b3f224e4fbd29961f07834179]) successfully deleted
Aug 10 19:19:26 dist-test-slave-7lbf krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:19:26 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:26.173981 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.156566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.7.43.62:38253)
0810 19:19:26.157198 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:26.157203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:26.157984 (+ 781us) server_negotiation.cc:407] Connection header received
0810 19:19:26.158167 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:26.158171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:26.158251 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:26.158348 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:26.160347 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.161166 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.163127 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.163589 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.163686 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.168906 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:26.168911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:26.168914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:26.168962 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:26.171611 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:26.172247 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:26.172251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:26.172253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:26.172372 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:26.172736 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:26.172739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:26.172742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:26.173034 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:26.173212 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:26.173465 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 19:19:26.173678 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
I20260810 19:19:26.177769 10245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:26.178427 10245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:26.185165 10701 tablet_service.cc:1467] Processing CreateTablet for tablet 34bcba28b3bf4da2a23303867a54e591 (DEFAULT_TABLE table=test-table [id=0a39cde256cc42baa6356ff7535451bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:26.185196 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 34bcba28b3bf4da2a23303867a54e591 (DEFAULT_TABLE table=test-table [id=0a39cde256cc42baa6356ff7535451bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:26.185398 10920 tablet_service.cc:1467] Processing CreateTablet for tablet 34bcba28b3bf4da2a23303867a54e591 (DEFAULT_TABLE table=test-table [id=0a39cde256cc42baa6356ff7535451bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:26.185655 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bcba28b3bf4da2a23303867a54e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:26.185709 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bcba28b3bf4da2a23303867a54e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:26.185937 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bcba28b3bf4da2a23303867a54e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:26.193531 11250 tablet_bootstrap.cc:492] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Bootstrap starting.
I20260810 19:19:26.194375 11251 tablet_bootstrap.cc:492] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Bootstrap starting.
I20260810 19:19:26.196458 11252 tablet_bootstrap.cc:492] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Bootstrap starting.
I20260810 19:19:26.196828 11250 tablet_bootstrap.cc:654] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:26.197046 11251 tablet_bootstrap.cc:654] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:26.200101 11252 tablet_bootstrap.cc:654] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:26.200440 11250 tablet_bootstrap.cc:492] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: No bootstrap required, opened a new log
I20260810 19:19:26.200625 11250 ts_tablet_manager.cc:1397] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260810 19:19:26.200790 11251 tablet_bootstrap.cc:492] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: No bootstrap required, opened a new log
I20260810 19:19:26.200999 11251 ts_tablet_manager.cc:1397] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 19:19:26.201763 11250 raft_consensus.cc:348] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.202183 11250 raft_consensus.cc:374] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:26.202173 11251 raft_consensus.cc:348] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.202387 11250 raft_consensus.cc:729] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d2627090cd49788c1bbacd8dce32a3, State: Initialized, Role: FOLLOWER
I20260810 19:19:26.202553 11251 raft_consensus.cc:374] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:26.202768 11251 raft_consensus.cc:729] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Initialized, Role: FOLLOWER
I20260810 19:19:26.202785 11250 consensus_queue.cc:260] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.203119 11251 consensus_queue.cc:260] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.204138 11251 ts_tablet_manager.cc:1428] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:26.204707 11250 ts_tablet_manager.cc:1428] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 19:19:26.205338 11252 tablet_bootstrap.cc:492] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: No bootstrap required, opened a new log
I20260810 19:19:26.205602 11252 ts_tablet_manager.cc:1397] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260810 19:19:26.207096 11252 raft_consensus.cc:348] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.207468 11252 raft_consensus.cc:374] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:26.207679 11252 raft_consensus.cc:729] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Initialized, Role: FOLLOWER
I20260810 19:19:26.208128 11252 consensus_queue.cc:260] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.209669 11252 ts_tablet_manager.cc:1428] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260810 19:19:26.309710 11266 raft_consensus.cc:482] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:26.310058 11266 raft_consensus.cc:504] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.312413 11266 leader_election.cc:288] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427), 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:26.323521 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.312577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:39159 (local address 127.7.43.2:34427)
0810 19:19:26.314420 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:26.314426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:26.314465 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:19:26.314531 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:26.314536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:26.314604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:26.315664 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:26.315965 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.317576 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.320607 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.321986 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.322104 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.322254 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:26.322841 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 19:19:26.323016 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1683,"thread_start_us":1479,"threads_started":1}
I20260810 19:19:26.323603 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.312514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:34427 (local address 127.7.43.1:39159)
0810 19:19:26.313650 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:26.313671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:26.313801 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:26.314743 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:26.314748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:26.314774 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:26.315488 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.315503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.317732 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.317737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.320407 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.320420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.322323 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.322331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.322667 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.322707 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:19:26.322809 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 19:19:26.322961 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":22,"thread_start_us":664,"threads_started":1}
I20260810 19:19:26.324554 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bcba28b3bf4da2a23303867a54e591" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3" is_pre_election: true
I20260810 19:19:26.324909 10734 raft_consensus.cc:2454] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 0.
I20260810 19:19:26.325608 10396 leader_election.cc:302] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 5899515cff904a8397fa4260be94ebc9; no voters:
I20260810 19:19:26.325809 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.313790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:60147 (local address 127.7.43.3:36587)
0810 19:19:26.315101 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:26.315108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:26.316236 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 19:19:26.316420 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:26.316430 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:26.316487 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:26.316567 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:26.317615 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.319227 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.322613 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.324300 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.324419 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.324561 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:26.325152 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 19:19:26.325320 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":834,"threads_started":1}
I20260810 19:19:26.325933 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.314153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.7.43.1:60147)
0810 19:19:26.316129 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:26.316143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:26.316293 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:26.316598 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:26.316602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:26.316626 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:26.317347 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.317357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.319400 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.319406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.322429 (+ 3023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.322443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.324641 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.324648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.324908 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.324942 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:19:26.325023 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:19:26.325143 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1813,"mutex_wait_us":42,"thread_start_us":1540,"threads_started":1}
I20260810 19:19:26.326164 11266 raft_consensus.cc:2781] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:26.326385 11266 raft_consensus.cc:482] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:26.326550 11266 raft_consensus.cc:3037] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:26.330245 10959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bcba28b3bf4da2a23303867a54e591" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a41b69697184a378c82e6ae3cb16a03" is_pre_election: true
I20260810 19:19:26.330598 10959 raft_consensus.cc:2454] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 0.
I20260810 19:19:26.331071 11266 raft_consensus.cc:504] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.332162 11266 leader_election.cc:288] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 election: Requested vote from peers 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427), 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:26.332551 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bcba28b3bf4da2a23303867a54e591" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3"
I20260810 19:19:26.332722 10959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bcba28b3bf4da2a23303867a54e591" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a41b69697184a378c82e6ae3cb16a03"
I20260810 19:19:26.332939 10734 raft_consensus.cc:3037] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:26.332965 10959 raft_consensus.cc:3037] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:26.335866 10959 raft_consensus.cc:2454] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 1.
I20260810 19:19:26.336275 10396 leader_election.cc:302] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a41b69697184a378c82e6ae3cb16a03, 5899515cff904a8397fa4260be94ebc9; no voters:
I20260810 19:19:26.336685 11266 raft_consensus.cc:2781] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:26.336826 10734 raft_consensus.cc:2454] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 1.
I20260810 19:19:26.337752 11266 raft_consensus.cc:686] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 1 LEADER]: Becoming Leader. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Running, Role: LEADER
I20260810 19:19:26.338385 11266 consensus_queue.cc:237] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.342710 10244 catalog_manager.cc:5495] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5899515cff904a8397fa4260be94ebc9 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "5899515cff904a8397fa4260be94ebc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } health_report { overall_health: UNKNOWN } } }
I20260810 19:19:26.401427 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.393246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.7.43.1:34341)
0810 19:19:26.394423 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:26.394427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:26.394461 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:19:26.394548 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:26.394551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:26.394623 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:26.394804 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:26.395441 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.396621 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.398450 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.399161 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.399274 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.399912 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:26.400616 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:26.400786 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:26.400839 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 19:19:26.400918 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003,"thread_start_us":780,"threads_started":1}
I20260810 19:19:26.420037 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.414062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:49839 (local address 127.7.43.3:36587)
0810 19:19:26.414275 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:26.414279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:26.414820 (+ 541us) server_negotiation.cc:407] Connection header received
0810 19:19:26.414986 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:26.414990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:26.415059 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:26.415132 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:26.415532 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.416238 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.418090 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.418852 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.418946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.419429 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:26.419545 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:26.419610 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:26.419702 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 19:19:26.419764 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:19:26.420074 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.413699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:36587 (local address 127.7.43.1:49839)
0810 19:19:26.414731 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:26.414751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:26.414876 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:26.415147 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:26.415150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:26.415159 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:26.415409 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.415416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.416342 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.416346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.417976 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.417990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.419072 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.419078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.419209 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.419267 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:26.419618 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:26.419626 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:19:26.419680 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 19:19:26.419778 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":694,"threads_started":1}
I20260810 19:19:26.428613 10959 raft_consensus.cc:1261] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Refusing update from remote peer 5899515cff904a8397fa4260be94ebc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:26.428733 10734 raft_consensus.cc:1261] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Refusing update from remote peer 5899515cff904a8397fa4260be94ebc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:26.429382 11272 consensus_queue.cc:1035] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:26.430056 11266 consensus_queue.cc:1035] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:26.462623 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.454159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:42443 (local address 127.7.43.1:34341)
0810 19:19:26.454310 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:26.454316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:26.455093 (+ 777us) server_negotiation.cc:407] Connection header received
0810 19:19:26.455277 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:26.455280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:26.455346 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:26.455415 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:26.456386 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.457622 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.459949 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.460939 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.461040 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.461778 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:26.461925 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:26.462025 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:26.462228 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 19:19:26.462295 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:19:26.462756 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:26.453793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34341 (local address 127.7.43.3:42443)
0810 19:19:26.454991 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:26.455009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:26.455158 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:26.455455 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:26.455459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:26.455470 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:19:26.456127 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.456140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.457847 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.457854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.459802 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:26.459818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:26.461185 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:26.461193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:26.461487 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:26.461565 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:19:26.462083 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:19:26.462095 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 19:19:26.462175 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:19:26.462319 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":710,"threads_started":1}
I20260810 19:19:26.495461 10245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:26.495749 10245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:26.499950 10245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Sending DeleteTablet for 3 replicas of tablet 34bcba28b3bf4da2a23303867a54e591
I20260810 19:19:26.501026 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bcba28b3bf4da2a23303867a54e591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:26 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59538
I20260810 19:19:26.501398 11232 tablet_replica.cc:330] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: stopping tablet replica
I20260810 19:19:26.501616 10920 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bcba28b3bf4da2a23303867a54e591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:26 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33608
I20260810 19:19:26.501740 11232 raft_consensus.cc:2229] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:26.501803 10701 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bcba28b3bf4da2a23303867a54e591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:26 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33726
I20260810 19:19:26.501921 11233 tablet_replica.cc:330] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: stopping tablet replica
I20260810 19:19:26.502202 11234 tablet_replica.cc:330] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: stopping tablet replica
I20260810 19:19:26.502202 11233 raft_consensus.cc:2229] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:26.502511 11233 raft_consensus.cc:2258] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:26.502494 11234 raft_consensus.cc:2229] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:26.502486 11232 raft_consensus.cc:2258] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:26.502779 11234 raft_consensus.cc:2258] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:26.503897 10245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 19:19:26.504122 11234 ts_tablet_manager.cc:1905] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:26.504242 11232 ts_tablet_manager.cc:1905] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 19:19:26.504889 10245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:26.510483 11232 ts_tablet_manager.cc:1918] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:26.510807 11232 log.cc:1196] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/34bcba28b3bf4da2a23303867a54e591
I20260810 19:19:26.511296 11232 ts_tablet_manager.cc:1939] T 34bcba28b3bf4da2a23303867a54e591 P 5899515cff904a8397fa4260be94ebc9: Deleting consensus metadata
I20260810 19:19:26.512456 10224 catalog_manager.cc:4841] TS 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341): tablet 34bcba28b3bf4da2a23303867a54e591 (table test-table [id=0a39cde256cc42baa6356ff7535451bd]) successfully deleted
I20260810 19:19:26.513485 11233 ts_tablet_manager.cc:1905] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:26.517684 10701 tablet_service.cc:1467] Processing CreateTablet for tablet 540227c30959458c9ee544aff1daa00c (DEFAULT_TABLE table=test-table [id=543d0998ab464b7a91a828f1678e8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:26.517931 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 540227c30959458c9ee544aff1daa00c (DEFAULT_TABLE table=test-table [id=543d0998ab464b7a91a828f1678e8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:26.518249 10701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 540227c30959458c9ee544aff1daa00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:26.518411 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 540227c30959458c9ee544aff1daa00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:26.519573 10920 tablet_service.cc:1467] Processing CreateTablet for tablet 540227c30959458c9ee544aff1daa00c (DEFAULT_TABLE table=test-table [id=543d0998ab464b7a91a828f1678e8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:26.520077 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 540227c30959458c9ee544aff1daa00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:26.526975 11252 tablet_bootstrap.cc:492] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Bootstrap starting.
I20260810 19:19:26.529203 11252 tablet_bootstrap.cc:654] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:26.531023 11233 ts_tablet_manager.cc:1918] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:26.531260 11233 log.cc:1196] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/34bcba28b3bf4da2a23303867a54e591
I20260810 19:19:26.531980 11233 ts_tablet_manager.cc:1939] T 34bcba28b3bf4da2a23303867a54e591 P 4a41b69697184a378c82e6ae3cb16a03: Deleting consensus metadata
I20260810 19:19:26.532933 11252 tablet_bootstrap.cc:492] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: No bootstrap required, opened a new log
I20260810 19:19:26.533221 11252 ts_tablet_manager.cc:1397] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260810 19:19:26.533270 11251 tablet_bootstrap.cc:492] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Bootstrap starting.
I20260810 19:19:26.533316 10220 catalog_manager.cc:4841] TS 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587): tablet 34bcba28b3bf4da2a23303867a54e591 (table test-table [id=0a39cde256cc42baa6356ff7535451bd]) successfully deleted
I20260810 19:19:26.534569 11252 raft_consensus.cc:348] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.534917 11252 raft_consensus.cc:374] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:26.535025 11252 raft_consensus.cc:729] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Initialized, Role: FOLLOWER
I20260810 19:19:26.535760 11251 tablet_bootstrap.cc:654] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:26.537123 11250 tablet_bootstrap.cc:492] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Bootstrap starting.
I20260810 19:19:26.535698 11252 consensus_queue.cc:260] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.539230 11252 ts_tablet_manager.cc:1428] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260810 19:19:26.539929 11250 tablet_bootstrap.cc:654] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:26.540606 11234 ts_tablet_manager.cc:1918] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:19:26.540904 11234 log.cc:1196] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/34bcba28b3bf4da2a23303867a54e591
I20260810 19:19:26.541859 11234 ts_tablet_manager.cc:1939] T 34bcba28b3bf4da2a23303867a54e591 P 07d2627090cd49788c1bbacd8dce32a3: Deleting consensus metadata
I20260810 19:19:26.543301 10220 catalog_manager.cc:4841] TS 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427): tablet 34bcba28b3bf4da2a23303867a54e591 (table test-table [id=0a39cde256cc42baa6356ff7535451bd]) successfully deleted
I20260810 19:19:26.555640 11250 tablet_bootstrap.cc:492] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: No bootstrap required, opened a new log
I20260810 19:19:26.555951 11250 ts_tablet_manager.cc:1397] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20260810 19:19:26.557346 11250 raft_consensus.cc:348] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.557930 11250 raft_consensus.cc:374] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:26.558106 11250 raft_consensus.cc:729] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d2627090cd49788c1bbacd8dce32a3, State: Initialized, Role: FOLLOWER
I20260810 19:19:26.558455 11250 consensus_queue.cc:260] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.558802 11251 tablet_bootstrap.cc:492] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: No bootstrap required, opened a new log
I20260810 19:19:26.559118 11251 ts_tablet_manager.cc:1397] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.007s
I20260810 19:19:26.561048 11251 raft_consensus.cc:348] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.561527 11251 raft_consensus.cc:374] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:26.561739 11251 raft_consensus.cc:729] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a41b69697184a378c82e6ae3cb16a03, State: Initialized, Role: FOLLOWER
I20260810 19:19:26.562342 11250 ts_tablet_manager.cc:1428] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260810 19:19:26.562254 11251 consensus_queue.cc:260] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.563047 11251 ts_tablet_manager.cc:1428] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 19:19:26.802106 11293 raft_consensus.cc:482] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:26.802433 11293 raft_consensus.cc:504] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.803493 11293 leader_election.cc:288] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427), 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:26.804018 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540227c30959458c9ee544aff1daa00c" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3" is_pre_election: true
I20260810 19:19:26.804179 10959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540227c30959458c9ee544aff1daa00c" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a41b69697184a378c82e6ae3cb16a03" is_pre_election: true
I20260810 19:19:26.804423 10959 raft_consensus.cc:2454] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 0.
I20260810 19:19:26.804491 10734 raft_consensus.cc:2454] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 0.
I20260810 19:19:26.805028 10396 leader_election.cc:302] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 5899515cff904a8397fa4260be94ebc9; no voters:
I20260810 19:19:26.805359 11293 raft_consensus.cc:2781] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:26.805610 11293 raft_consensus.cc:482] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:26.805783 11293 raft_consensus.cc:3037] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:26.809844 11293 raft_consensus.cc:504] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.810784 11293 leader_election.cc:288] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 election: Requested vote from peers 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427), 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587)
I20260810 19:19:26.811209 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540227c30959458c9ee544aff1daa00c" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d2627090cd49788c1bbacd8dce32a3"
I20260810 19:19:26.811554 10734 raft_consensus.cc:3037] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:26.811585 10959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540227c30959458c9ee544aff1daa00c" candidate_uuid: "5899515cff904a8397fa4260be94ebc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a41b69697184a378c82e6ae3cb16a03"
I20260810 19:19:26.811879 10959 raft_consensus.cc:3037] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:26.814510 10734 raft_consensus.cc:2454] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 1.
I20260810 19:19:26.815230 10396 leader_election.cc:302] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d2627090cd49788c1bbacd8dce32a3, 5899515cff904a8397fa4260be94ebc9; no voters:
I20260810 19:19:26.815660 11293 raft_consensus.cc:2781] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:26.815836 10959 raft_consensus.cc:2454] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5899515cff904a8397fa4260be94ebc9 in term 1.
I20260810 19:19:26.815941 11293 raft_consensus.cc:686] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 1 LEADER]: Becoming Leader. State: Replica: 5899515cff904a8397fa4260be94ebc9, State: Running, Role: LEADER
I20260810 19:19:26.816582 11293 consensus_queue.cc:237] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } }
I20260810 19:19:26.820622 10244 catalog_manager.cc:5495] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5899515cff904a8397fa4260be94ebc9 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "5899515cff904a8397fa4260be94ebc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5899515cff904a8397fa4260be94ebc9" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 } health_report { overall_health: UNKNOWN } } }
I20260810 19:19:26.855293 10959 raft_consensus.cc:1261] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Refusing update from remote peer 5899515cff904a8397fa4260be94ebc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:26.855571 10734 raft_consensus.cc:1261] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Refusing update from remote peer 5899515cff904a8397fa4260be94ebc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:26.856177 11296 consensus_queue.cc:1035] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a41b69697184a378c82e6ae3cb16a03" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:26.856547 11293 consensus_queue.cc:1035] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d2627090cd49788c1bbacd8dce32a3" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:19:26.890676 11206 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 19:19:28.487417 10244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:28.487738 10244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:28.493880 10244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9fb43d5c504f57aedc97a8e6189631: Sending DeleteTablet for 3 replicas of tablet 540227c30959458c9ee544aff1daa00c
I20260810 19:19:28.494884 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 540227c30959458c9ee544aff1daa00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:28 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59538
I20260810 19:19:28.494979 10701 tablet_service.cc:1514] Processing DeleteTablet for tablet 540227c30959458c9ee544aff1daa00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:28 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33726
I20260810 19:19:28.495002 10920 tablet_service.cc:1514] Processing DeleteTablet for tablet 540227c30959458c9ee544aff1daa00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:28 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33608
I20260810 19:19:28.497874 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10370
I20260810 19:19:28.507783 11344 tablet_replica.cc:330] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: stopping tablet replica
I20260810 19:19:28.508240 11344 raft_consensus.cc:2229] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:28.508867 11344 raft_consensus.cc:2258] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:28.511654 11345 tablet_replica.cc:330] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: stopping tablet replica
I20260810 19:19:28.511781 11347 tablet_replica.cc:330] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: stopping tablet replica
I20260810 19:19:28.512032 11345 raft_consensus.cc:2229] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:28.512055 11347 raft_consensus.cc:2229] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:28.512161 11344 ts_tablet_manager.cc:1905] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:28.512470 11347 raft_consensus.cc:2258] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:28.512507 11345 raft_consensus.cc:2258] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:28.513504 11347 ts_tablet_manager.cc:1905] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:28.513510 11345 ts_tablet_manager.cc:1905] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:28.520648 11344 ts_tablet_manager.cc:1918] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:19:28.520848 11344 log.cc:1196] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/540227c30959458c9ee544aff1daa00c
I20260810 19:19:28.521322 11345 ts_tablet_manager.cc:1918] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:19:28.521533 11344 ts_tablet_manager.cc:1939] T 540227c30959458c9ee544aff1daa00c P 5899515cff904a8397fa4260be94ebc9: Deleting consensus metadata
I20260810 19:19:28.521665 11345 log.cc:1196] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/540227c30959458c9ee544aff1daa00c
I20260810 19:19:28.522009 11347 ts_tablet_manager.cc:1918] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 19:19:28.522341 11347 log.cc:1196] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/540227c30959458c9ee544aff1daa00c
I20260810 19:19:28.522994 11345 ts_tablet_manager.cc:1939] T 540227c30959458c9ee544aff1daa00c P 4a41b69697184a378c82e6ae3cb16a03: Deleting consensus metadata
I20260810 19:19:28.523129 10224 catalog_manager.cc:4841] TS 5899515cff904a8397fa4260be94ebc9 (127.7.43.1:34341): tablet 540227c30959458c9ee544aff1daa00c (table test-table [id=543d0998ab464b7a91a828f1678e8186]) successfully deleted
I20260810 19:19:28.523218 11347 ts_tablet_manager.cc:1939] T 540227c30959458c9ee544aff1daa00c P 07d2627090cd49788c1bbacd8dce32a3: Deleting consensus metadata
I20260810 19:19:28.524369 10220 catalog_manager.cc:4841] TS 07d2627090cd49788c1bbacd8dce32a3 (127.7.43.2:34427): tablet 540227c30959458c9ee544aff1daa00c (table test-table [id=543d0998ab464b7a91a828f1678e8186]) successfully deleted
I20260810 19:19:28.524633 10220 catalog_manager.cc:4841] TS 4a41b69697184a378c82e6ae3cb16a03 (127.7.43.3:36587): tablet 540227c30959458c9ee544aff1daa00c (table test-table [id=543d0998ab464b7a91a828f1678e8186]) successfully deleted
Aug 10 19:19:28 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:28.536717 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:28.499480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.7.43.1:34341)
0810 19:19:28.509942 (+ 10462us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:28.509947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:28.510003 (+ 56us) server_negotiation.cc:407] Connection header received
0810 19:19:28.510443 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:28.510450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:28.510574 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:28.510665 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:28.512785 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:28.514486 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:28.516753 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:28.517759 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:28.517899 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:28.529222 (+ 11323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:28.529240 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:28.529258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:28.529296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:28.533044 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:28.533873 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:28.533882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:28.533884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:28.534095 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:28.534703 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:28.534707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:28.534710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:28.535142 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:28.535652 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:28.535876 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 19:19:28.536183 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10302,"thread_start_us":10217,"threads_started":1}
W20260810 19:19:28.537554 10580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42518
W20260810 19:19:28.538425 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10370: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:28.538662 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10370
I20260810 19:19:28.574651 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10595
Aug 10 19:19:28 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:28.590243 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:28.575501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.7.43.2:34427)
0810 19:19:28.576040 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:28.576044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:28.576060 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:19:28.576114 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:28.576118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:28.576176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:28.576271 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:28.578047 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:28.578781 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:28.580072 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:28.580570 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:28.580663 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:28.585097 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:28.585112 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:28.585129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:28.585160 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:28.587837 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:28.588452 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:28.588462 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:28.588466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:28.588626 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:28.588894 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:28.588897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:28.588899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:28.589231 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:28.589449 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:28.589726 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 19:19:28.589931 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":275,"threads_started":1}
W20260810 19:19:28.591034 10797 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678
W20260810 19:19:28.591614 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10595: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:28.591794 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10595
I20260810 19:19:28.624644 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10816
Aug 10 19:19:28 dist-test-slave-7lbf krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389566, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:28.639101 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:28.625349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.7.43.3:36587)
0810 19:19:28.625873 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:28.625877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:28.625893 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:19:28.625958 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:28.625961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:28.626020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:28.626110 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:28.627703 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:28.628436 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:28.629666 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:28.630081 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:28.630182 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:28.634155 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:28.634170 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:28.634187 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:28.634219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:28.636746 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:28.637336 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:28.637344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:28.637346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:28.637460 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:28.637803 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:28.637807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:28.637809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:28.638163 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:28.638393 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:28.638601 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 19:19:28.638766 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
W20260810 19:19:28.639777 11036 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54222
W20260810 19:19:28.640443 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:28.640605 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10816
I20260810 19:19:28.674165 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10195
I20260810 19:19:28.675433 10325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:28.793805 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10195
2026-08-10T19:19:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9355 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:28 dist-test-slave-7lbf krb5kdc[11359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:28 dist-test-slave-7lbf krb5kdc[11359](info): set up 2 sockets
Aug 10 19:19:28 dist-test-slave-7lbf krb5kdc[11359](info): commencing operation
krb5kdc: starting...
W20260810 19:19:30.025197 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:30 dist-test-slave-7lbf krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:30.248309 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36541
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:30.574311 11385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:30.574594 11385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:30.574684 11385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:30.581930 11385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:30.582036 11385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:30.582088 11385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:30.582144 11385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:30.582191 11385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:30.592136 11385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:30.593477 11385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:30.594815 11385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:30.606333 11393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:30.606376 11395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:30.606303 11392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:30.607234 11385 server_base.cc:1034] running on GCE node
I20260810 19:19:30.608054 11385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:30.609468 11385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:30.610724 11385 hybrid_clock.cc:648] HybridClock initialized: now 1786389570610668 us; error 42 us; skew 500 ppm
I20260810 19:19:30.611088 11385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:30 dist-test-slave-7lbf krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389570, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:30.614276 11385 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:30.615693 11385 webserver.cc:460] Webserver started at http://127.7.43.62:46873/ using document root <none> and password file <none>
I20260810 19:19:30.616362 11385 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:30.616467 11385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:30.616778 11385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:30.619755 11385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "38d7ac9349b8405ea0664221c86890f8"
format_stamp: "Formatted at 2026-08-10 19:19:30 on dist-test-slave-7lbf"
server_key: "bd2f057f6da532b89aec4c292d6d653b"
server_key_iv: "86801f2d4d76f546aa3a7cfe68b8336b"
server_key_version: "encryptionkey@0"
I20260810 19:19:30.620501 11385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "38d7ac9349b8405ea0664221c86890f8"
format_stamp: "Formatted at 2026-08-10 19:19:30 on dist-test-slave-7lbf"
server_key: "bd2f057f6da532b89aec4c292d6d653b"
server_key_iv: "86801f2d4d76f546aa3a7cfe68b8336b"
server_key_version: "encryptionkey@0"
I20260810 19:19:30.625624 11385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:19:30.629148 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:30.630554 11385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:19:30.630716 11385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "38d7ac9349b8405ea0664221c86890f8"
format_stamp: "Formatted at 2026-08-10 19:19:30 on dist-test-slave-7lbf"
server_key: "bd2f057f6da532b89aec4c292d6d653b"
server_key_iv: "86801f2d4d76f546aa3a7cfe68b8336b"
server_key_version: "encryptionkey@0"
I20260810 19:19:30.630906 11385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:30.669770 11385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:30.673795 11385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:30.674077 11385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:30.691358 11385 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:36541
I20260810 19:19:30.691388 11492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:36541 every 8 connection(s)
I20260810 19:19:30.692710 11385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:30.696465 11493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:30.698439 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11385
I20260810 19:19:30.698650 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:30.699254 7340 external_mini_cluster.cc:1428] Setting key 97052f55478f1892b0c6660307474f11
I20260810 19:19:30.708521 11493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8: Bootstrap starting.
I20260810 19:19:30.712380 11493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:30.713752 11493 log.cc:824] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:19:30 dist-test-slave-7lbf krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:30.716722 11493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8: No bootstrap required, opened a new log
I20260810 19:19:30.722401 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:30.701237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.7.43.62:36541)
0810 19:19:30.702063 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:30.702075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:30.702117 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:19:30.703808 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:30.703831 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:30.704231 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:30.704575 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:30.706463 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:30.708115 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:30.709879 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:30.710597 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:30.710737 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:30.715398 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:30.715406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:30.715420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:30.715465 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:30.718432 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:30.719107 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:30.719113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:30.719119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:30.719280 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:30.719659 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:30.719662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:30.719664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:30.720190 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:30.720471 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:30.720958 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 19:19:30.721336 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":383,"threads_started":1}
I20260810 19:19:30.724661 11493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7ac9349b8405ea0664221c86890f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36541 } }
I20260810 19:19:30.725091 11493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:30.725203 11493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d7ac9349b8405ea0664221c86890f8, State: Initialized, Role: FOLLOWER
I20260810 19:19:30.725755 11493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7ac9349b8405ea0664221c86890f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36541 } }
I20260810 19:19:30.725997 11493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:30.726116 11493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:30.726263 11493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:30.728361 11493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7ac9349b8405ea0664221c86890f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36541 } }
I20260810 19:19:30.728816 11493 leader_election.cc:302] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38d7ac9349b8405ea0664221c86890f8; no voters:
I20260810 19:19:30.729373 11493 leader_election.cc:288] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:30.729540 11502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:30.730187 11502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [term 1 LEADER]: Becoming Leader. State: Replica: 38d7ac9349b8405ea0664221c86890f8, State: Running, Role: LEADER
I20260810 19:19:30.730718 11502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7ac9349b8405ea0664221c86890f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36541 } }
I20260810 19:19:30.731555 11493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:30.733322 11505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38d7ac9349b8405ea0664221c86890f8. Latest consensus state: current_term: 1 leader_uuid: "38d7ac9349b8405ea0664221c86890f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7ac9349b8405ea0664221c86890f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36541 } } }
I20260810 19:19:30.733772 11505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:30.733348 11504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38d7ac9349b8405ea0664221c86890f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d7ac9349b8405ea0664221c86890f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36541 } } }
I20260810 19:19:30.735103 11504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:30.735690 11516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:30.740075 11516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:30.749014 11516 catalog_manager.cc:1347] Generated new cluster ID: 0be5e06949314b6a80b3cba82d2e8a9a
I20260810 19:19:30.749114 11516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:30.765728 11516 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:30.767046 11516 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:30.774751 11516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38d7ac9349b8405ea0664221c86890f8: Generated new TSK 0
I20260810 19:19:30.775826 11516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:30.832530 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36541
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:31.165136 11539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:31.165429 11539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:31.165551 11539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:31.172787 11539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:31.172894 11539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:31.173012 11539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:31.182711 11539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:31.183933 11539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:31.185177 11539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:31.198029 11555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:31.198127 11551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:31.198295 11557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:31.198799 11539 server_base.cc:1034] running on GCE node
I20260810 19:19:31.199399 11539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:31.200350 11539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:31.201592 11539 hybrid_clock.cc:648] HybridClock initialized: now 1786389571201529 us; error 46 us; skew 500 ppm
I20260810 19:19:31.201916 11539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:31 dist-test-slave-7lbf krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:31.205080 11539 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:31.206532 11539 webserver.cc:460] Webserver started at http://127.7.43.1:41791/ using document root <none> and password file <none>
I20260810 19:19:31.207201 11539 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:31.207304 11539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:31.207572 11539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:31.210515 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "09fb38943ee64d1392255c8b23562c07"
format_stamp: "Formatted at 2026-08-10 19:19:31 on dist-test-slave-7lbf"
server_key: "76a2d439bbb8150a5b1ebef24250494a"
server_key_iv: "9e1eafd28e1b934ef08af5315c535ce8"
server_key_version: "encryptionkey@0"
I20260810 19:19:31.211288 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "09fb38943ee64d1392255c8b23562c07"
format_stamp: "Formatted at 2026-08-10 19:19:31 on dist-test-slave-7lbf"
server_key: "76a2d439bbb8150a5b1ebef24250494a"
server_key_iv: "9e1eafd28e1b934ef08af5315c535ce8"
server_key_version: "encryptionkey@0"
I20260810 19:19:31.217320 11539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 19:19:31.220952 11567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:31.222569 11539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:31.222731 11539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "09fb38943ee64d1392255c8b23562c07"
format_stamp: "Formatted at 2026-08-10 19:19:31 on dist-test-slave-7lbf"
server_key: "76a2d439bbb8150a5b1ebef24250494a"
server_key_iv: "9e1eafd28e1b934ef08af5315c535ce8"
server_key_version: "encryptionkey@0"
I20260810 19:19:31.222985 11539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:31.236912 11539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:31.241272 11539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:31.241595 11539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:31.242581 11539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:31.244063 11539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:31.244144 11539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:31.244266 11539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:31.244323 11539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:31.274688 11539 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:44495
I20260810 19:19:31.274757 11763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:44495 every 8 connection(s)
I20260810 19:19:31.275957 11539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:31.284454 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11539
I20260810 19:19:31.284641 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:31.285271 7340 external_mini_cluster.cc:1428] Setting key 5c88fe1391923f20713494d8687a6360
Aug 10 19:19:31 dist-test-slave-7lbf krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:31.295758 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:31.278447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:60033 (local address 127.7.43.62:36541)
0810 19:19:31.278852 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:31.278857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:31.279757 (+ 900us) server_negotiation.cc:407] Connection header received
0810 19:19:31.281573 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:31.281579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:31.281656 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:31.281750 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:31.283210 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.283910 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.285488 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.285995 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.286119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:31.289889 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:31.289896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:31.289901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:31.289941 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:31.292867 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:31.293574 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:31.293579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:31.293582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:31.293721 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:31.294118 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:31.294122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:31.294125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:31.294446 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:31.294570 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:31.295230 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 19:19:31.295440 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":212,"threads_started":1}
I20260810 19:19:31.296620 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:31.278891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36541 (local address 127.7.43.1:60033)
0810 19:19:31.279596 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:31.279639 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:31.281284 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:31.281988 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:31.281995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:31.282498 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:31.283041 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:31.283058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.284075 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.284079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:31.285350 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:31.285364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.286249 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.286256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:31.286522 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:31.287148 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:31.287175 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:31.289655 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:31.293016 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:31.293023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:31.293036 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:31.293439 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:31.293821 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:31.293825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:31.293827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:31.294024 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:31.294549 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:31.294554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:31.294869 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 19:19:31.295104 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 19:19:31.295488 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":275,"threads_started":1}
I20260810 19:19:31.298136 11765 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36541
I20260810 19:19:31.298465 11765 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:31.299292 11765 heartbeater.cc:511] Master 127.7.43.62:36541 requested a full tablet report, sending...
I20260810 19:19:31.301584 11434 ts_manager.cc:194] Registered new tserver with Master: 09fb38943ee64d1392255c8b23562c07 (127.7.43.1:44495)
I20260810 19:19:31.303048 11434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:60033
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:31.337266 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36541
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:31.663928 11772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:31.664211 11772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:31.664331 11772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:31.671608 11772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:31.671717 11772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:31.671867 11772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:31.681421 11772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:31.682677 11772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:31.683945 11772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:31.696446 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:31.696874 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:31.697103 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:31.697364 11772 server_base.cc:1034] running on GCE node
I20260810 19:19:31.697990 11772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:31.698949 11772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:31.700163 11772 hybrid_clock.cc:648] HybridClock initialized: now 1786389571700110 us; error 45 us; skew 500 ppm
I20260810 19:19:31.700467 11772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:31 dist-test-slave-7lbf krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:31.703644 11772 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:31.705049 11772 webserver.cc:460] Webserver started at http://127.7.43.2:46393/ using document root <none> and password file <none>
I20260810 19:19:31.705737 11772 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:31.705858 11772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:31.706154 11772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:31.709002 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "b32c1f89ca3f4d5aaff1bcba1ea04992"
format_stamp: "Formatted at 2026-08-10 19:19:31 on dist-test-slave-7lbf"
server_key: "13da0b0400b2ab3f39dba4b2be596545"
server_key_iv: "d1361888b75524cca44b9bfcc2d19da5"
server_key_version: "encryptionkey@0"
I20260810 19:19:31.709811 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "b32c1f89ca3f4d5aaff1bcba1ea04992"
format_stamp: "Formatted at 2026-08-10 19:19:31 on dist-test-slave-7lbf"
server_key: "13da0b0400b2ab3f39dba4b2be596545"
server_key_iv: "d1361888b75524cca44b9bfcc2d19da5"
server_key_version: "encryptionkey@0"
I20260810 19:19:31.714888 11772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 19:19:31.718211 11799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:31.719465 11772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:31.719614 11772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "b32c1f89ca3f4d5aaff1bcba1ea04992"
format_stamp: "Formatted at 2026-08-10 19:19:31 on dist-test-slave-7lbf"
server_key: "13da0b0400b2ab3f39dba4b2be596545"
server_key_iv: "d1361888b75524cca44b9bfcc2d19da5"
server_key_version: "encryptionkey@0"
I20260810 19:19:31.719785 11772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:31.736538 11772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:31.740396 11772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:31.740653 11772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:31.741602 11772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:31.742923 11772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:31.743007 11772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:31.743130 11772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:31.743184 11772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:31.773932 11772 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:37155
I20260810 19:19:31.773970 11990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:37155 every 8 connection(s)
I20260810 19:19:31.775203 11772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:31.779655 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11772
I20260810 19:19:31.779804 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:31.780262 7340 external_mini_cluster.cc:1428] Setting key 39f0212e2a98811513f18e9894734f6f
Aug 10 19:19:31 dist-test-slave-7lbf krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:31.795135 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:31.777619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:60493 (local address 127.7.43.62:36541)
0810 19:19:31.777824 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:31.777830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:31.779018 (+ 1188us) server_negotiation.cc:407] Connection header received
0810 19:19:31.780937 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:31.780941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:31.781029 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:31.781132 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:31.782937 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.783868 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.785331 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.785943 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.786065 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:31.789257 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:31.789263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:31.789267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:31.789305 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:31.791949 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:31.792772 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:31.792776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:31.792778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:31.792912 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:31.793392 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:31.793397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:31.793400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:31.793748 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:31.793864 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:31.794719 (+ 855us) server_negotiation.cc:299] Negotiation successful
0810 19:19:31.794890 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:19:31.796558 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:31.778161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36541 (local address 127.7.43.2:60493)
0810 19:19:31.778854 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:31.778899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:31.780634 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:31.781362 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:31.781374 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:31.782122 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:31.782770 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:31.782788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.783994 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.783998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:31.785192 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:31.785204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:31.786142 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:31.786147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:31.786345 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:31.786962 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:31.786989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:31.789041 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:31.792092 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:31.792101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:31.792118 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:31.792641 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:31.793031 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:31.793036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:31.793039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:31.793274 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:31.793874 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:31.793881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:31.794297 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 19:19:31.794669 (+ 372us) client_negotiation.cc:240] Negotiation successful
0810 19:19:31.795198 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":277,"threads_started":1}
I20260810 19:19:31.798055 11991 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36541
I20260810 19:19:31.798346 11991 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:31.799058 11991 heartbeater.cc:511] Master 127.7.43.62:36541 requested a full tablet report, sending...
I20260810 19:19:31.800431 11434 ts_manager.cc:194] Registered new tserver with Master: b32c1f89ca3f4d5aaff1bcba1ea04992 (127.7.43.2:37155)
I20260810 19:19:31.801204 11434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:60493
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:31.832231 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36541
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:32.161079 12003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:32.161374 12003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:32.161499 12003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:32.168792 12003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:32.168905 12003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:32.169065 12003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:32.178823 12003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:32.180107 12003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:32.181406 12003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:32.198323 12013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:32.198419 12014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:32.198884 12017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:32.199594 12003 server_base.cc:1034] running on GCE node
I20260810 19:19:32.200206 12003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:32.201233 12003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:32.202453 12003 hybrid_clock.cc:648] HybridClock initialized: now 1786389572202390 us; error 56 us; skew 500 ppm
I20260810 19:19:32.202801 12003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:32 dist-test-slave-7lbf krb5kdc[11359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:32.205997 12003 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:32.207512 12003 webserver.cc:460] Webserver started at http://127.7.43.3:36717/ using document root <none> and password file <none>
I20260810 19:19:32.208221 12003 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:32.208334 12003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:32.208657 12003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:32.211777 12003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "366162758421498b9d66e923f5434c9a"
format_stamp: "Formatted at 2026-08-10 19:19:32 on dist-test-slave-7lbf"
server_key: "18a069d24061c312e657e17049b191a2"
server_key_iv: "21e2fc0ae0844dee330625c65bd06783"
server_key_version: "encryptionkey@0"
I20260810 19:19:32.212548 12003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "366162758421498b9d66e923f5434c9a"
format_stamp: "Formatted at 2026-08-10 19:19:32 on dist-test-slave-7lbf"
server_key: "18a069d24061c312e657e17049b191a2"
server_key_iv: "21e2fc0ae0844dee330625c65bd06783"
server_key_version: "encryptionkey@0"
I20260810 19:19:32.218199 12003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:19:32.222083 12029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:32.223682 12003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:19:32.223850 12003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "366162758421498b9d66e923f5434c9a"
format_stamp: "Formatted at 2026-08-10 19:19:32 on dist-test-slave-7lbf"
server_key: "18a069d24061c312e657e17049b191a2"
server_key_iv: "21e2fc0ae0844dee330625c65bd06783"
server_key_version: "encryptionkey@0"
I20260810 19:19:32.224078 12003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:32.238745 12003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:32.243165 12003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:32.243445 12003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:32.244481 12003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:32.245945 12003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:32.246030 12003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:32.246151 12003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:32.246204 12003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:32.281085 12003 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:37833
I20260810 19:19:32.281147 12199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:37833 every 8 connection(s)
I20260810 19:19:32.282495 12003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:32.286334 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12003
I20260810 19:19:32.286515 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:32.287101 7340 external_mini_cluster.cc:1428] Setting key 328a43f86a4be938cc7dcb5a639bbb88
Aug 10 19:19:32 dist-test-slave-7lbf krb5kdc[11359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389572, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:32.305035 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:32.285088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:35953 (local address 127.7.43.62:36541)
0810 19:19:32.285300 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:32.285306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:32.286574 (+ 1268us) server_negotiation.cc:407] Connection header received
0810 19:19:32.288684 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:32.288690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:32.288780 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:32.288884 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:32.290893 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:32.292241 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:32.293866 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:32.294519 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:32.294641 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:32.298341 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:32.298348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:32.298353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:32.298398 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:32.301805 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:32.302764 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:32.302770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:32.302774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:32.302944 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:32.303369 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:32.303374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:32.303378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:32.303735 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:32.303883 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:32.304530 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 19:19:32.304707 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:19:32.305986 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:32.285622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36541 (local address 127.7.43.3:35953)
0810 19:19:32.286417 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:32.286461 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:32.288381 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:32.289167 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:32.289180 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:32.289914 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:32.290725 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:32.290745 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:32.292400 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:32.292404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:32.293713 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:32.293726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:32.294714 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:32.294719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:32.294916 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:32.295703 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:32.295726 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:32.298102 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:32.302130 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:32.302140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:32.302157 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:32.302631 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:32.303063 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:32.303067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:32.303069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:32.303258 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:32.303839 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:32.303845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:32.304182 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 19:19:32.304406 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 19:19:32.304796 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":320,"threads_started":1}
I20260810 19:19:32.306540 11765 heartbeater.cc:503] Master 127.7.43.62:36541 was elected leader, sending a full tablet report...
I20260810 19:19:32.307629 12203 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36541
I20260810 19:19:32.308040 12203 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:32.309034 12203 heartbeater.cc:511] Master 127.7.43.62:36541 requested a full tablet report, sending...
I20260810 19:19:32.310707 11434 ts_manager.cc:194] Registered new tserver with Master: 366162758421498b9d66e923f5434c9a (127.7.43.3:37833)
I20260810 19:19:32.311510 11434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:35953
I20260810 19:19:32.315040 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 19:19:32.364013 11768 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:32.360707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.7.43.62:36541)
0810 19:19:32.360951 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:32.360958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:32.361369 (+ 411us) server_negotiation.cc:407] Connection header received
0810 19:19:32.361622 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:32.361628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:32.361717 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:32.361889 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:32.363592 (+ 1703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42284: BlockingRecv error: recv got EOF from 127.0.0.1:42284 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:19:32.367419 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11539
W20260810 19:19:32.370481 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11539: Not authorized: Client connection negotiation failed: client connection to 127.7.43.1:44495: server requires authentication, but client does not have Kerberos credentials available
I20260810 19:19:32.370702 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11539
I20260810 19:19:32.419582 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11772
W20260810 19:19:32.422606 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11772: Not authorized: Client connection negotiation failed: client connection to 127.7.43.2:37155: server requires authentication, but client does not have Kerberos credentials available
I20260810 19:19:32.422767 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11772
I20260810 19:19:32.455513 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12003
W20260810 19:19:32.459072 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12003: Not authorized: Client connection negotiation failed: client connection to 127.7.43.3:37833: server requires authentication, but client does not have Kerberos credentials available
I20260810 19:19:32.459299 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12003
W20260810 19:19:32.459371 12225 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:32.456780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.7.43.3:37833)
0810 19:19:32.457459 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:32.457466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:32.457496 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:19:32.457591 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:32.457598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:32.457811 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:32.457954 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:32.459018 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49376: BlockingRecv error: recv got EOF from 127.0.0.1:49376 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
I20260810 19:19:32.497831 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11385
I20260810 19:19:32.499091 11488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:32.602206 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11385
2026-08-10T19:19:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3811 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:32 dist-test-slave-7lbf krb5kdc[12235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:32 dist-test-slave-7lbf krb5kdc[12235](info): set up 2 sockets
Aug 10 19:19:32 dist-test-slave-7lbf krb5kdc[12235](info): commencing operation
krb5kdc: starting...
W20260810 19:19:33.850752 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:34 dist-test-slave-7lbf krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:34.077927 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:37305
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:34.406762 12254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:34.407088 12254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:34.407228 12254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:34.414601 12254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:34.414737 12254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:34.414820 12254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:34.414902 12254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:34.414968 12254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:34.424983 12254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:37305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:34.426307 12254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:34.427634 12254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:34.439316 12263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:34.439236 12264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:34.439283 12270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:34.440299 12254 server_base.cc:1034] running on GCE node
I20260810 19:19:34.441098 12254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:34.442652 12254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:34.443861 12254 hybrid_clock.cc:648] HybridClock initialized: now 1786389574443814 us; error 49 us; skew 500 ppm
I20260810 19:19:34.444207 12254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:34 dist-test-slave-7lbf krb5kdc[12235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389574, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:34.450655 12254 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:34.452308 12254 webserver.cc:460] Webserver started at http://127.7.43.62:39863/ using document root <none> and password file <none>
I20260810 19:19:34.453032 12254 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:34.453155 12254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:34.453464 12254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:34.456564 12254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "fc4484feb0184b73a84aefb5ab2b7249"
format_stamp: "Formatted at 2026-08-10 19:19:34 on dist-test-slave-7lbf"
server_key: "329b2672c827d2ae093f8c2012fdde3b"
server_key_iv: "955f9431ce5e69ed68e9d755d80f91a7"
server_key_version: "encryptionkey@0"
I20260810 19:19:34.457378 12254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "fc4484feb0184b73a84aefb5ab2b7249"
format_stamp: "Formatted at 2026-08-10 19:19:34 on dist-test-slave-7lbf"
server_key: "329b2672c827d2ae093f8c2012fdde3b"
server_key_iv: "955f9431ce5e69ed68e9d755d80f91a7"
server_key_version: "encryptionkey@0"
I20260810 19:19:34.462775 12254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:19:34.466419 12280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:34.467805 12254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:34.467978 12254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "fc4484feb0184b73a84aefb5ab2b7249"
format_stamp: "Formatted at 2026-08-10 19:19:34 on dist-test-slave-7lbf"
server_key: "329b2672c827d2ae093f8c2012fdde3b"
server_key_iv: "955f9431ce5e69ed68e9d755d80f91a7"
server_key_version: "encryptionkey@0"
I20260810 19:19:34.468173 12254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:34.483860 12254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:34.488121 12254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:34.488394 12254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:34.506001 12254 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:37305
I20260810 19:19:34.506047 12356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:37305 every 8 connection(s)
I20260810 19:19:34.507473 12254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:34.509140 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12254
I20260810 19:19:34.509371 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:34.510236 7340 external_mini_cluster.cc:1428] Setting key 18b10c58e20df8842315a60a38d7f411
I20260810 19:19:34.512198 12357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:34.523614 12357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249: Bootstrap starting.
Aug 10 19:19:34 dist-test-slave-7lbf krb5kdc[12235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:34.527025 12357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:34.528105 12357 log.cc:824] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:34.530965 12357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249: No bootstrap required, opened a new log
I20260810 19:19:34.531838 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:34.512225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.7.43.62:37305)
0810 19:19:34.513025 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:34.513038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:34.513078 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:19:34.514840 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:34.514864 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:34.515275 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:34.515729 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:34.517231 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:34.518601 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:34.519825 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:34.520530 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:34.520718 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:34.524726 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:34.524734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:34.524749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:34.524785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:34.527770 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:34.528459 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:34.528466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:34.528472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:34.528710 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:34.529116 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:34.529120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:34.529127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:34.529724 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:34.529990 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:34.530468 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 19:19:34.530860 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":354,"threads_started":1}
I20260810 19:19:34.535660 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:19:34.538991 12357 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4484feb0184b73a84aefb5ab2b7249" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37305 } }
I20260810 19:19:34.539322 12357 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:34.539417 12357 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4484feb0184b73a84aefb5ab2b7249, State: Initialized, Role: FOLLOWER
I20260810 19:19:34.539922 12357 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4484feb0184b73a84aefb5ab2b7249" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37305 } }
I20260810 19:19:34.540156 12357 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:34.540289 12357 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:34.540446 12357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:34.542593 12357 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4484feb0184b73a84aefb5ab2b7249" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37305 } }
I20260810 19:19:34.543081 12357 leader_election.cc:302] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4484feb0184b73a84aefb5ab2b7249; no voters:
I20260810 19:19:34.543684 12357 leader_election.cc:288] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:34.543792 12366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:34.545252 12366 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [term 1 LEADER]: Becoming Leader. State: Replica: fc4484feb0184b73a84aefb5ab2b7249, State: Running, Role: LEADER
I20260810 19:19:34.545972 12357 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:34.545885 12366 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4484feb0184b73a84aefb5ab2b7249" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37305 } }
I20260810 19:19:34.549852 12368 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4484feb0184b73a84aefb5ab2b7249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4484feb0184b73a84aefb5ab2b7249" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37305 } } }
I20260810 19:19:34.550212 12368 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:34.550733 12369 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4484feb0184b73a84aefb5ab2b7249. Latest consensus state: current_term: 1 leader_uuid: "fc4484feb0184b73a84aefb5ab2b7249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4484feb0184b73a84aefb5ab2b7249" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37305 } } }
I20260810 19:19:34.551035 12369 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:34.566978 12381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:34.572587 12381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:34.581990 12381 catalog_manager.cc:1347] Generated new cluster ID: 51e70971e7014fe88a2f04cb8a9e879b
I20260810 19:19:34.582183 12381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:34.594815 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:34.584530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.7.43.62:37305)
0810 19:19:34.584696 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:34.584703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:34.585369 (+ 666us) server_negotiation.cc:407] Connection header received
0810 19:19:34.585715 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:34.585721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:34.585804 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:34.585881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:34.587448 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:34.588628 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:34.590602 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:34.591249 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:34.591350 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:34.592791 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:34.592798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:34.592803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:34.593906 (+ 1103us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:34.594010 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:34.594049 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:34.594162 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:34.594274 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 19:19:34.594380 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 19:19:34.613770 12381 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:34.615662 12381 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:34.624881 12381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc4484feb0184b73a84aefb5ab2b7249: Generated new TSK 0
I20260810 19:19:34.625762 12381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:19:34.641072 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12254
I20260810 19:19:34.642557 12351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:34.741595 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12254
2026-08-10T19:19:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2125 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:34 dist-test-slave-7lbf krb5kdc[12416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:34 dist-test-slave-7lbf krb5kdc[12416](info): set up 2 sockets
Aug 10 19:19:34 dist-test-slave-7lbf krb5kdc[12416](info): commencing operation
krb5kdc: starting...
W20260810 19:19:35.969234 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:36 dist-test-slave-7lbf krb5kdc[12416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:36.205286 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43441
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:43441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:36.531553 12443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:36.531832 12443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:36.531965 12443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:36.539021 12443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:36.539135 12443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:36.539212 12443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:36.539284 12443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:36.539348 12443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:36.549041 12443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:43441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:36.550383 12443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:36.551733 12443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:36.563159 12449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:36.563081 12450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:36.563108 12452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:36.563920 12443 server_base.cc:1034] running on GCE node
I20260810 19:19:36.564683 12443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:36.566119 12443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:36.567329 12443 hybrid_clock.cc:648] HybridClock initialized: now 1786389576567281 us; error 40 us; skew 500 ppm
I20260810 19:19:36.567658 12443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:36 dist-test-slave-7lbf krb5kdc[12416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389576, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:36.570695 12443 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:36.572098 12443 webserver.cc:460] Webserver started at http://127.7.43.62:40893/ using document root <none> and password file <none>
I20260810 19:19:36.572759 12443 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:36.572877 12443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:36.573169 12443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:36.576124 12443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "8d6d9ab2245e4a27b913889d7a220c96"
format_stamp: "Formatted at 2026-08-10 19:19:36 on dist-test-slave-7lbf"
server_key: "1414bafed9f233a954f5c79468eb17ef"
server_key_iv: "ed62140890e5dd3797ae3a8cad05c6ba"
server_key_version: "encryptionkey@0"
I20260810 19:19:36.576882 12443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "8d6d9ab2245e4a27b913889d7a220c96"
format_stamp: "Formatted at 2026-08-10 19:19:36 on dist-test-slave-7lbf"
server_key: "1414bafed9f233a954f5c79468eb17ef"
server_key_iv: "ed62140890e5dd3797ae3a8cad05c6ba"
server_key_version: "encryptionkey@0"
I20260810 19:19:36.581933 12443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 19:19:36.585276 12472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:36.586621 12443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:36.586782 12443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "8d6d9ab2245e4a27b913889d7a220c96"
format_stamp: "Formatted at 2026-08-10 19:19:36 on dist-test-slave-7lbf"
server_key: "1414bafed9f233a954f5c79468eb17ef"
server_key_iv: "ed62140890e5dd3797ae3a8cad05c6ba"
server_key_version: "encryptionkey@0"
I20260810 19:19:36.586959 12443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:36.612268 12443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:36.616653 12443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:36.616931 12443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:36.634415 12443 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:43441
I20260810 19:19:36.634459 12542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:43441 every 8 connection(s)
I20260810 19:19:36.635756 12443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:36.636787 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12443
I20260810 19:19:36.637045 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:36.637889 7340 external_mini_cluster.cc:1428] Setting key 3e3e90d4f3d819837edfedbe42c13dc5
I20260810 19:19:36.640671 12543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:19:36 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:36.653990 12543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96: Bootstrap starting.
I20260810 19:19:36.658196 12543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:36.659473 12543 log.cc:824] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:36.662593 12543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96: No bootstrap required, opened a new log
I20260810 19:19:36.662724 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:36.640010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.7.43.62:43441)
0810 19:19:36.640814 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:36.640829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:36.640880 (+ 51us) server_negotiation.cc:407] Connection header received
0810 19:19:36.643187 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:36.643217 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:36.643834 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:36.644308 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:36.645856 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:36.647906 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:36.649238 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:36.650298 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:36.650511 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:36.654605 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:36.654615 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:36.654636 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:36.654690 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:36.658500 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:36.659201 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:36.659211 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:36.659220 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:36.659451 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:36.659794 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:36.659798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:36.659802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:36.660505 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:36.660825 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:36.661290 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 19:19:36.661735 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":334,"threads_started":1}
I20260810 19:19:36.670684 12543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6d9ab2245e4a27b913889d7a220c96" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43441 } }
I20260810 19:19:36.671067 12543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:36.671162 12543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6d9ab2245e4a27b913889d7a220c96, State: Initialized, Role: FOLLOWER
I20260810 19:19:36.671690 12543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6d9ab2245e4a27b913889d7a220c96" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43441 } }
I20260810 19:19:36.671909 12543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:36.672041 12543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:36.672199 12543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:36.674579 12543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6d9ab2245e4a27b913889d7a220c96" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43441 } }
I20260810 19:19:36.675062 12543 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6d9ab2245e4a27b913889d7a220c96; no voters:
I20260810 19:19:36.675659 12543 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:36.675797 12550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:36.676539 12550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6d9ab2245e4a27b913889d7a220c96, State: Running, Role: LEADER
I20260810 19:19:36.677117 12550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6d9ab2245e4a27b913889d7a220c96" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43441 } }
I20260810 19:19:36.677676 12543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:36.680167 12552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6d9ab2245e4a27b913889d7a220c96. Latest consensus state: current_term: 1 leader_uuid: "8d6d9ab2245e4a27b913889d7a220c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6d9ab2245e4a27b913889d7a220c96" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43441 } } }
I20260810 19:19:36.680132 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6d9ab2245e4a27b913889d7a220c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6d9ab2245e4a27b913889d7a220c96" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43441 } } }
I20260810 19:19:36.680552 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:36.680564 12552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:36.681480 12563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:36.686863 12563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:36.696298 12563 catalog_manager.cc:1347] Generated new cluster ID: 884b0b8c4f86417395753745766ae203
I20260810 19:19:36.696426 12563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:36.757234 12563 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:36.759512 12563 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:36.782477 12563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6d9ab2245e4a27b913889d7a220c96: Generated new TSK 0
I20260810 19:19:36.783598 12563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:36.844503 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43441
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:37.174546 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:37.174870 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:37.174998 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:37.182518 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:37.182626 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:37.182745 12588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:37.192552 12588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:37.193907 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:37.195214 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:37.208209 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:37.208238 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:37.208173 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:37.208868 12588 server_base.cc:1034] running on GCE node
I20260810 19:19:37.209411 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:37.210350 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:37.211540 12588 hybrid_clock.cc:648] HybridClock initialized: now 1786389577211479 us; error 56 us; skew 500 ppm
I20260810 19:19:37.211807 12588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:37 dist-test-slave-7lbf krb5kdc[12416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389577, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:37.214891 12588 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:37.216293 12588 webserver.cc:460] Webserver started at http://127.7.43.1:33057/ using document root <none> and password file <none>
I20260810 19:19:37.216946 12588 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:37.217046 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:37.217303 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:37.220156 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "784c118aeebf4720a298ababbbdeeeea"
format_stamp: "Formatted at 2026-08-10 19:19:37 on dist-test-slave-7lbf"
server_key: "7c49eb905f8820de9896051780cf6300"
server_key_iv: "4c2f82978fdc972ae89f63902c2b36d9"
server_key_version: "encryptionkey@0"
I20260810 19:19:37.220904 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "784c118aeebf4720a298ababbbdeeeea"
format_stamp: "Formatted at 2026-08-10 19:19:37 on dist-test-slave-7lbf"
server_key: "7c49eb905f8820de9896051780cf6300"
server_key_iv: "4c2f82978fdc972ae89f63902c2b36d9"
server_key_version: "encryptionkey@0"
I20260810 19:19:37.226009 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:19:37.231153 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:37.232491 12588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20260810 19:19:37.232648 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "784c118aeebf4720a298ababbbdeeeea"
format_stamp: "Formatted at 2026-08-10 19:19:37 on dist-test-slave-7lbf"
server_key: "7c49eb905f8820de9896051780cf6300"
server_key_iv: "4c2f82978fdc972ae89f63902c2b36d9"
server_key_version: "encryptionkey@0"
I20260810 19:19:37.232839 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:37.248539 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:37.252454 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:37.252717 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:37.253652 12588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:37.254992 12588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:37.255070 12588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:37.255187 12588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:37.255244 12588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:37.285737 12588 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:45035
I20260810 19:19:37.285787 12816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:45035 every 8 connection(s)
I20260810 19:19:37.287029 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:37.297096 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12588
I20260810 19:19:37.297310 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:37.298038 7340 external_mini_cluster.cc:1428] Setting key 5663c1ba75a20af4b2bc2f3daae5492a
Aug 10 19:19:37 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389577, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:37.306949 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:37.289459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:55777 (local address 127.7.43.62:43441)
0810 19:19:37.289919 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:37.289923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:37.290876 (+ 953us) server_negotiation.cc:407] Connection header received
0810 19:19:37.292736 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:37.292742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:37.292822 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:37.292919 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:37.294544 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.295261 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.296833 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.297272 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.297390 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:37.301009 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:37.301015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:37.301019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:37.301056 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:37.304010 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:37.304707 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:37.304712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:37.304714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:37.304859 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:37.305246 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:37.305250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:37.305253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:37.305603 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:37.305736 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:37.306408 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 19:19:37.306628 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":250,"threads_started":1}
I20260810 19:19:37.307775 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:37.289957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43441 (local address 127.7.43.1:55777)
0810 19:19:37.290710 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:37.290752 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:37.292466 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:37.293177 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:37.293185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:37.293757 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:37.294354 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:37.294373 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.295428 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.295431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:37.296684 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:37.296697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.297545 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.297553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:37.297777 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:37.298438 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:37.298472 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:37.300792 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:37.304154 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:37.304162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:37.304177 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:37.304585 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:37.304958 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:37.304961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:37.304963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:37.305142 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:37.305703 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:37.305709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:37.306043 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 19:19:37.306269 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 19:19:37.306652 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":273,"threads_started":1}
I20260810 19:19:37.309413 12817 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43441
I20260810 19:19:37.309784 12817 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:37.310736 12817 heartbeater.cc:511] Master 127.7.43.62:43441 requested a full tablet report, sending...
I20260810 19:19:37.312762 12491 ts_manager.cc:194] Registered new tserver with Master: 784c118aeebf4720a298ababbbdeeeea (127.7.43.1:45035)
I20260810 19:19:37.314234 12491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:55777
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:37.348264 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43441
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:37.689683 12826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:37.690055 12826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:37.690219 12826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:37.697647 12826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:37.697806 12826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:37.697986 12826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:37.708004 12826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:37.709394 12826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:37.710819 12826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:37.724359 12834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:37.724334 12835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:37.724561 12839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:37.725063 12826 server_base.cc:1034] running on GCE node
I20260810 19:19:37.725710 12826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:37.726711 12826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:37.727948 12826 hybrid_clock.cc:648] HybridClock initialized: now 1786389577727889 us; error 45 us; skew 500 ppm
I20260810 19:19:37.728269 12826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:37 dist-test-slave-7lbf krb5kdc[12416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389577, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:37.731688 12826 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:37.733155 12826 webserver.cc:460] Webserver started at http://127.7.43.2:43619/ using document root <none> and password file <none>
I20260810 19:19:37.733871 12826 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:37.733997 12826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:37.734292 12826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:37.737795 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "f402ea4e845e482ba326c593251f79dd"
format_stamp: "Formatted at 2026-08-10 19:19:37 on dist-test-slave-7lbf"
server_key: "d0bfd5950a69877674cf4ea999edc1ea"
server_key_iv: "49f128f8d0426b47b5cd42e3a7c97ee8"
server_key_version: "encryptionkey@0"
I20260810 19:19:37.738571 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "f402ea4e845e482ba326c593251f79dd"
format_stamp: "Formatted at 2026-08-10 19:19:37 on dist-test-slave-7lbf"
server_key: "d0bfd5950a69877674cf4ea999edc1ea"
server_key_iv: "49f128f8d0426b47b5cd42e3a7c97ee8"
server_key_version: "encryptionkey@0"
I20260810 19:19:37.743774 12826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:37.747136 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:37.748461 12826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:19:37.748615 12826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "f402ea4e845e482ba326c593251f79dd"
format_stamp: "Formatted at 2026-08-10 19:19:37 on dist-test-slave-7lbf"
server_key: "d0bfd5950a69877674cf4ea999edc1ea"
server_key_iv: "49f128f8d0426b47b5cd42e3a7c97ee8"
server_key_version: "encryptionkey@0"
I20260810 19:19:37.748791 12826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:37.771443 12826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:37.775806 12826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:37.776103 12826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:37.777069 12826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:37.778578 12826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:37.778666 12826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:37.778791 12826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:37.778846 12826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:37.810050 12826 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:39301
I20260810 19:19:37.810096 13044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:39301 every 8 connection(s)
I20260810 19:19:37.811328 12826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:37.819881 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12826
I20260810 19:19:37.820087 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:37.820647 7340 external_mini_cluster.cc:1428] Setting key fa95ffbf2043ad5c5ee56483b3c7ebc0
Aug 10 19:19:37 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389577, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:37.831254 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:37.813759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:49611 (local address 127.7.43.62:43441)
0810 19:19:37.814170 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:37.814175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:37.815105 (+ 930us) server_negotiation.cc:407] Connection header received
0810 19:19:37.816890 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:37.816896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:37.816983 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:37.817080 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:37.818592 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.819303 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.821456 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.821950 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.822064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:37.825828 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:37.825833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:37.825836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:37.825868 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:37.828311 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:37.829087 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:37.829091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:37.829093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:37.829217 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:37.829673 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:37.829675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:37.829677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:37.829939 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:37.830046 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:37.830807 (+ 761us) server_negotiation.cc:299] Negotiation successful
0810 19:19:37.830996 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
I20260810 19:19:37.832376 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:37.814236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43441 (local address 127.7.43.2:49611)
0810 19:19:37.814937 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:37.814986 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:37.816622 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:37.817321 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:37.817329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:37.817864 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:37.818420 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:37.818442 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.819448 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.819451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:37.821313 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:37.821325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:37.822165 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:37.822172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:37.822397 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:37.823005 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:37.823034 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:37.825602 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:37.828442 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:37.828451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:37.828467 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:37.828984 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:37.829310 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:37.829314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:37.829317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:37.829587 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:37.830049 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:37.830056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:37.830441 (+ 385us) client_negotiation.cc:769] Sending connection context
0810 19:19:37.830735 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 19:19:37.831228 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":267,"threads_started":1}
I20260810 19:19:37.833674 13046 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43441
I20260810 19:19:37.834018 13046 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:37.834870 13046 heartbeater.cc:511] Master 127.7.43.62:43441 requested a full tablet report, sending...
I20260810 19:19:37.836367 12491 ts_manager.cc:194] Registered new tserver with Master: f402ea4e845e482ba326c593251f79dd (127.7.43.2:39301)
I20260810 19:19:37.837203 12491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:49611
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:37.872723 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43441
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:38.198480 13056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:38.198766 13056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:38.198912 13056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:38.206070 13056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:38.206182 13056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:38.206331 13056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:38.216096 13056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:38.217404 13056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:38.218765 13056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:38.231508 13062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:38.231544 13063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:38.231571 13067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:38.232278 13056 server_base.cc:1034] running on GCE node
I20260810 19:19:38.232810 13056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:38.233701 13056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:38.234896 13056 hybrid_clock.cc:648] HybridClock initialized: now 1786389578234843 us; error 47 us; skew 500 ppm
I20260810 19:19:38.235189 13056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:38.238304 13056 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:38.239670 13056 webserver.cc:460] Webserver started at http://127.7.43.3:45215/ using document root <none> and password file <none>
I20260810 19:19:38.240335 13056 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:38.240458 13056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:38.240756 13056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:38.243847 13056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "972a8e07a0ed411f94a1b7cff369440e"
format_stamp: "Formatted at 2026-08-10 19:19:38 on dist-test-slave-7lbf"
server_key: "94e8ff129337fd71881ef59aafe5ea50"
server_key_iv: "c119604fe3f376c8bcfe1b13669fb8a5"
server_key_version: "encryptionkey@0"
I20260810 19:19:38.244640 13056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "972a8e07a0ed411f94a1b7cff369440e"
format_stamp: "Formatted at 2026-08-10 19:19:38 on dist-test-slave-7lbf"
server_key: "94e8ff129337fd71881ef59aafe5ea50"
server_key_iv: "c119604fe3f376c8bcfe1b13669fb8a5"
server_key_version: "encryptionkey@0"
I20260810 19:19:38.249656 13056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 19:19:38.253055 13074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:38.254647 13056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:38.254797 13056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "972a8e07a0ed411f94a1b7cff369440e"
format_stamp: "Formatted at 2026-08-10 19:19:38 on dist-test-slave-7lbf"
server_key: "94e8ff129337fd71881ef59aafe5ea50"
server_key_iv: "c119604fe3f376c8bcfe1b13669fb8a5"
server_key_version: "encryptionkey@0"
I20260810 19:19:38.254987 13056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:38.284106 13056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:38.288378 13056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:38.288678 13056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:38.289710 13056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:38.291179 13056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:38.291252 13056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:38.291375 13056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:38.291426 13056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:38.317206 12817 heartbeater.cc:503] Master 127.7.43.62:43441 was elected leader, sending a full tablet report...
I20260810 19:19:38.321854 13056 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:42285
I20260810 19:19:38.321938 13253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:42285 every 8 connection(s)
I20260810 19:19:38.323114 13056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:38.324098 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13056
I20260810 19:19:38.324304 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:38.324911 7340 external_mini_cluster.cc:1428] Setting key bec2d538b91dd75ba234dfb085cfc07a
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:38.343034 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:38.326042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:57609 (local address 127.7.43.62:43441)
0810 19:19:38.326221 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:38.326227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:38.327635 (+ 1408us) server_negotiation.cc:407] Connection header received
0810 19:19:38.329532 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:38.329537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:38.329614 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:38.329691 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:38.331112 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.331840 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.333437 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.333901 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.334007 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:38.337186 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:38.337191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:38.337194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:38.337233 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:38.339927 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.340737 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.340742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.340746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.340864 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.341283 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.341288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.341292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.341616 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:38.341755 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:38.342464 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 19:19:38.342694 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:19:38.344098 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:38.326653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43441 (local address 127.7.43.3:57609)
0810 19:19:38.327454 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:38.327505 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:38.329279 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:38.329889 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:38.329896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:38.330403 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:38.330983 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:38.331000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.331962 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.331965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:38.333330 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:38.333342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.334089 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.334094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:38.334292 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:38.334907 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:38.334939 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:38.336996 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:38.340062 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:38.340071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:38.340088 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:38.340623 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:38.340976 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:38.340981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:38.340985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:38.341175 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:38.341751 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:38.341759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:38.342100 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 19:19:38.342346 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 19:19:38.342895 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":315,"threads_started":1}
I20260810 19:19:38.345377 13254 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43441
I20260810 19:19:38.345702 13254 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:38.346437 13254 heartbeater.cc:511] Master 127.7.43.62:43441 requested a full tablet report, sending...
I20260810 19:19:38.347822 12491 ts_manager.cc:194] Registered new tserver with Master: 972a8e07a0ed411f94a1b7cff369440e (127.7.43.3:42285)
I20260810 19:19:38.348642 12491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:57609
I20260810 19:19:38.352661 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:38.410462 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:38.396812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.7.43.62:43441)
0810 19:19:38.397011 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:38.397017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:38.397403 (+ 386us) server_negotiation.cc:407] Connection header received
0810 19:19:38.397604 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:38.397608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:38.397675 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:38.397764 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:38.399158 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.399928 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.401226 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.401723 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.401838 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:38.405555 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:38.405563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:38.405568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:38.405609 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:38.407868 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.408475 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.408481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.408484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.408608 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.408985 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.408990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.408993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.409321 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:38.409618 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:38.409883 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 19:19:38.410130 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 19:19:38.411120 12491 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46856
I20260810 19:19:38.412756 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12588
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:38.429039 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:38.413784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.7.43.1:45035)
0810 19:19:38.414404 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:38.414411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:38.414436 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:19:38.414493 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:38.414499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:38.414702 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:38.414821 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:38.415859 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.416880 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.418139 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.418957 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.419081 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:38.423374 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:38.423391 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:38.423407 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:38.423448 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:38.426410 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.427096 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.427105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.427107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.427248 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.427575 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.427578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.427580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.428029 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:38.428274 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:38.428468 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 19:19:38.428706 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
W20260810 19:19:38.430433 12806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57994
W20260810 19:19:38.431591 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:38.431768 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12588
I20260810 19:19:38.452060 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12826
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:38.468159 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:38.452953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.7.43.2:39301)
0810 19:19:38.453594 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:38.453602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:38.453628 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:19:38.453691 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:38.453698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:38.453909 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:38.454038 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:38.455243 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.456264 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.457729 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.458310 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.458405 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:38.462808 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:38.462821 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:38.462836 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:38.462880 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:38.465691 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.466286 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.466296 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.466298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.466417 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.466708 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.466711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.466713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.467076 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:38.467307 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:38.467586 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 19:19:38.467794 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":303,"threads_started":1}
W20260810 19:19:38.469642 13038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39630
W20260810 19:19:38.470824 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:38.471012 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12826
I20260810 19:19:38.504458 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13056
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[12416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389578, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:38.520802 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:38.505613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.7.43.3:42285)
0810 19:19:38.506346 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:38.506353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:38.506383 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:19:38.506443 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:38.506449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:38.506657 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:38.506783 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:38.508053 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.509085 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.510318 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:38.510922 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:38.511025 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:38.515203 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:38.515215 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:38.515230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:38.515263 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:38.518021 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.518746 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.518753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.518756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.518879 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:38.519230 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:38.519234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:38.519236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:38.519584 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:38.519895 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:38.520200 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 19:19:38.520468 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":411,"threads_started":1}
W20260810 19:19:38.530886 13246 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60158
W20260810 19:19:38.532121 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13056: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:19:38.532290 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13056
I20260810 19:19:38.551980 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12443
I20260810 19:19:38.553190 12535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:38.656903 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12443
2026-08-10T19:19:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3922 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[13279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[13279](info): set up 2 sockets
Aug 10 19:19:38 dist-test-slave-7lbf krb5kdc[13279](info): commencing operation
krb5kdc: starting...
W20260810 19:19:39.900241 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:40 dist-test-slave-7lbf krb5kdc[13279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:40.128820 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:41155
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:40.457679 13299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:40.458019 13299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:40.458150 13299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:40.465359 13299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:40.465495 13299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:40.465596 13299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:40.465674 13299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:40.465737 13299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:40.475911 13299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:41155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:40.477293 13299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:40.478652 13299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:40.489910 13308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:40.489872 13311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:40.489961 13313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:40.491082 13299 server_base.cc:1034] running on GCE node
I20260810 19:19:40.491860 13299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:40.493307 13299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:40.494513 13299 hybrid_clock.cc:648] HybridClock initialized: now 1786389580494440 us; error 55 us; skew 500 ppm
I20260810 19:19:40.494858 13299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:40 dist-test-slave-7lbf krb5kdc[13279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389580, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:40.503721 13299 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:40.505321 13299 webserver.cc:460] Webserver started at http://127.7.43.62:46341/ using document root <none> and password file <none>
I20260810 19:19:40.506121 13299 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:40.506249 13299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:40.506564 13299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:40.509740 13299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "b748bcae48ca4a70b7ff4e49df4402e2"
format_stamp: "Formatted at 2026-08-10 19:19:40 on dist-test-slave-7lbf"
server_key: "279d2bfed2d140c96a62dec7494c4232"
server_key_iv: "f9376067596d91c6a651204ce033a7f8"
server_key_version: "encryptionkey@0"
I20260810 19:19:40.510550 13299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "b748bcae48ca4a70b7ff4e49df4402e2"
format_stamp: "Formatted at 2026-08-10 19:19:40 on dist-test-slave-7lbf"
server_key: "279d2bfed2d140c96a62dec7494c4232"
server_key_iv: "f9376067596d91c6a651204ce033a7f8"
server_key_version: "encryptionkey@0"
I20260810 19:19:40.515748 13299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:40.519202 13323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:40.520527 13299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:19:40.520694 13299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "b748bcae48ca4a70b7ff4e49df4402e2"
format_stamp: "Formatted at 2026-08-10 19:19:40 on dist-test-slave-7lbf"
server_key: "279d2bfed2d140c96a62dec7494c4232"
server_key_iv: "f9376067596d91c6a651204ce033a7f8"
server_key_version: "encryptionkey@0"
I20260810 19:19:40.520882 13299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:40.536845 13299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:40.540663 13299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:40.540926 13299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:40.558571 13299 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:41155
I20260810 19:19:40.558594 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:41155 every 8 connection(s)
I20260810 19:19:40.559914 13299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:40.560374 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13299
I20260810 19:19:40.560581 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:40.561373 7340 external_mini_cluster.cc:1428] Setting key 0db701d4f8fb6ae34048f4ed63666818
I20260810 19:19:40.564461 13417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:19:40 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:40.577021 13417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2: Bootstrap starting.
I20260810 19:19:40.580160 13417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:40.581164 13417 log.cc:824] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:40.584005 13417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2: No bootstrap required, opened a new log
I20260810 19:19:40.584061 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:40.563449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.7.43.62:41155)
0810 19:19:40.564276 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:40.564292 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:40.564342 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:19:40.566647 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:40.566675 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:40.567168 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:40.567630 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:40.568926 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:40.571029 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:40.572213 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:40.573194 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:40.573378 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:40.577295 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:40.577304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:40.577319 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:40.577360 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:40.580232 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:40.580888 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:40.580899 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:40.580906 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:40.581067 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:40.581440 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:40.581445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:40.581447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:40.582006 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:40.582304 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:40.582727 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 19:19:40.583120 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":351,"threads_started":1}
I20260810 19:19:40.591959 13417 raft_consensus.cc:348] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41155 } }
I20260810 19:19:40.592322 13417 raft_consensus.cc:374] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:40.592417 13417 raft_consensus.cc:729] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b748bcae48ca4a70b7ff4e49df4402e2, State: Initialized, Role: FOLLOWER
I20260810 19:19:40.592973 13417 consensus_queue.cc:260] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41155 } }
I20260810 19:19:40.593206 13417 raft_consensus.cc:388] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:40.593328 13417 raft_consensus.cc:482] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:40.593499 13417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:40.595654 13417 raft_consensus.cc:504] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41155 } }
I20260810 19:19:40.596136 13417 leader_election.cc:302] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b748bcae48ca4a70b7ff4e49df4402e2; no voters:
I20260810 19:19:40.596709 13417 leader_election.cc:288] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:40.596825 13426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:40.597966 13426 raft_consensus.cc:686] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [term 1 LEADER]: Becoming Leader. State: Replica: b748bcae48ca4a70b7ff4e49df4402e2, State: Running, Role: LEADER
I20260810 19:19:40.598588 13426 consensus_queue.cc:237] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41155 } }
I20260810 19:19:40.598968 13417 sys_catalog.cc:564] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:40.601382 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41155 } } }
I20260810 19:19:40.601818 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:40.601691 13429 sys_catalog.cc:455] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b748bcae48ca4a70b7ff4e49df4402e2. Latest consensus state: current_term: 1 leader_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b748bcae48ca4a70b7ff4e49df4402e2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41155 } } }
I20260810 19:19:40.602008 13429 sys_catalog.cc:458] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:40.602849 13437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:40.608942 13437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:40.617443 13437 catalog_manager.cc:1347] Generated new cluster ID: 590cd6c995334138afd4c421fa830eb7
I20260810 19:19:40.617574 13437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:40.645124 13437 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:40.647096 13437 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:40.661263 13437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b748bcae48ca4a70b7ff4e49df4402e2: Generated new TSK 0
I20260810 19:19:40.662369 13437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:40.720665 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:41155
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:41.055996 13453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:41.056344 13453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:41.056502 13453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:41.063786 13453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:41.063911 13453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:41.064065 13453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:41.074029 13453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:41.075354 13453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:41.076673 13453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:41.089562 13461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:41.089622 13459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:41.089599 13463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:41.090480 13453 server_base.cc:1034] running on GCE node
I20260810 19:19:41.091147 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:41.092103 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:41.093307 13453 hybrid_clock.cc:648] HybridClock initialized: now 1786389581093247 us; error 53 us; skew 500 ppm
I20260810 19:19:41.093662 13453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:41 dist-test-slave-7lbf krb5kdc[13279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389581, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:41.096738 13453 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:41.098239 13453 webserver.cc:460] Webserver started at http://127.7.43.1:46143/ using document root <none> and password file <none>
I20260810 19:19:41.098984 13453 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:41.099110 13453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:41.099407 13453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:41.102476 13453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "30f73dec3fc743e0a8dc6adad1060d5c"
format_stamp: "Formatted at 2026-08-10 19:19:41 on dist-test-slave-7lbf"
server_key: "2c1276738343af4cd471bc8b5b813568"
server_key_iv: "9c46b1135943f9feebc9f0e616c7c6e0"
server_key_version: "encryptionkey@0"
I20260810 19:19:41.103276 13453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "30f73dec3fc743e0a8dc6adad1060d5c"
format_stamp: "Formatted at 2026-08-10 19:19:41 on dist-test-slave-7lbf"
server_key: "2c1276738343af4cd471bc8b5b813568"
server_key_iv: "9c46b1135943f9feebc9f0e616c7c6e0"
server_key_version: "encryptionkey@0"
I20260810 19:19:41.108609 13453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:19:41.112131 13474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:41.113456 13453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:19:41.113663 13453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "30f73dec3fc743e0a8dc6adad1060d5c"
format_stamp: "Formatted at 2026-08-10 19:19:41 on dist-test-slave-7lbf"
server_key: "2c1276738343af4cd471bc8b5b813568"
server_key_iv: "9c46b1135943f9feebc9f0e616c7c6e0"
server_key_version: "encryptionkey@0"
I20260810 19:19:41.113860 13453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:41.132826 13453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:41.136827 13453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:41.137127 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:41.138130 13453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:41.139510 13453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:41.139590 13453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:41.139719 13453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:41.139775 13453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:41.170892 13453 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:46685
I20260810 19:19:41.170946 13670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:46685 every 8 connection(s)
I20260810 19:19:41.172186 13453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:41.175706 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13453
I20260810 19:19:41.175884 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:41.176759 7340 external_mini_cluster.cc:1428] Setting key 06385c59a9698566fe5b96a171ab1f42
Aug 10 19:19:41 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389581, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:41.194796 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:41.174983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:44337 (local address 127.7.43.62:41155)
0810 19:19:41.175438 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:41.175445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:41.176705 (+ 1260us) server_negotiation.cc:407] Connection header received
0810 19:19:41.178940 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:41.178947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:41.179050 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:41.179149 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:41.181188 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.181958 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.184151 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.184549 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.184665 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:41.188721 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:41.188728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:41.188733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:41.188773 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:41.191833 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:41.192515 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:41.192521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:41.192524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:41.192688 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:41.193064 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:41.193068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:41.193071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:41.193385 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:41.193551 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:41.194238 (+ 687us) server_negotiation.cc:299] Negotiation successful
0810 19:19:41.194467 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":200,"threads_started":1}
I20260810 19:19:41.195616 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:41.175592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:41155 (local address 127.7.43.1:44337)
0810 19:19:41.176507 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:41.176564 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:41.178679 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:41.179433 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:41.179456 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:41.180187 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:41.181018 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:41.181045 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.182112 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.182117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:41.184009 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:41.184024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.184787 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.184794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:41.185072 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:41.185803 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:41.185838 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:41.188471 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:41.191963 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:41.191970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:41.191983 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:41.192390 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:41.192781 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:41.192783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:41.192786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:41.192967 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:41.193499 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:41.193505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:41.193856 (+ 351us) client_negotiation.cc:769] Sending connection context
0810 19:19:41.194090 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 19:19:41.194490 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":339,"threads_started":1}
I20260810 19:19:41.197141 13671 heartbeater.cc:347] Connected to a master server at 127.7.43.62:41155
I20260810 19:19:41.197499 13671 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:41.198416 13671 heartbeater.cc:511] Master 127.7.43.62:41155 requested a full tablet report, sending...
I20260810 19:19:41.200914 13353 ts_manager.cc:194] Registered new tserver with Master: 30f73dec3fc743e0a8dc6adad1060d5c (127.7.43.1:46685)
I20260810 19:19:41.202401 13353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:44337
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:41.232916 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:41155
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:41.561683 13680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:41.561972 13680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:41.562098 13680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:41.569280 13680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:41.569391 13680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:41.569563 13680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:41.579418 13680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:41.580669 13680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:41.582021 13680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:41.594769 13688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:41.594717 13691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:41.594794 13689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:41.595561 13680 server_base.cc:1034] running on GCE node
I20260810 19:19:41.596134 13680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:41.597020 13680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:41.598210 13680 hybrid_clock.cc:648] HybridClock initialized: now 1786389581598160 us; error 40 us; skew 500 ppm
I20260810 19:19:41.598515 13680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:41 dist-test-slave-7lbf krb5kdc[13279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389581, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:41.601567 13680 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:41.603008 13680 webserver.cc:460] Webserver started at http://127.7.43.2:36849/ using document root <none> and password file <none>
I20260810 19:19:41.603668 13680 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:41.603772 13680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:41.604038 13680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:41.607038 13680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "a1fcea8222a5432eaeaba9cc19bd43a5"
format_stamp: "Formatted at 2026-08-10 19:19:41 on dist-test-slave-7lbf"
server_key: "91842d19e7942122b4cbf5c51d7399b5"
server_key_iv: "d65c03ea42795b508b95727c49da267d"
server_key_version: "encryptionkey@0"
I20260810 19:19:41.607791 13680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "a1fcea8222a5432eaeaba9cc19bd43a5"
format_stamp: "Formatted at 2026-08-10 19:19:41 on dist-test-slave-7lbf"
server_key: "91842d19e7942122b4cbf5c51d7399b5"
server_key_iv: "d65c03ea42795b508b95727c49da267d"
server_key_version: "encryptionkey@0"
I20260810 19:19:41.612824 13680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 19:19:41.616375 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:41.617745 13680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:41.617897 13680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "a1fcea8222a5432eaeaba9cc19bd43a5"
format_stamp: "Formatted at 2026-08-10 19:19:41 on dist-test-slave-7lbf"
server_key: "91842d19e7942122b4cbf5c51d7399b5"
server_key_iv: "d65c03ea42795b508b95727c49da267d"
server_key_version: "encryptionkey@0"
I20260810 19:19:41.618084 13680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:41.676859 13680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:41.680778 13680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:41.681066 13680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:41.682049 13680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:41.683348 13680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:41.683425 13680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:41.683537 13680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:41.683591 13680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:41.713451 13680 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45433
I20260810 19:19:41.713508 13893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45433 every 8 connection(s)
I20260810 19:19:41.714707 13680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:41.716080 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13680
I20260810 19:19:41.716235 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:41.716670 7340 external_mini_cluster.cc:1428] Setting key bbae0733cdbe0b089ee1dfef3759b39f
Aug 10 19:19:41 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389581, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:41.735658 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:41.717227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:58277 (local address 127.7.43.62:41155)
0810 19:19:41.717752 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:41.717758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:41.718951 (+ 1193us) server_negotiation.cc:407] Connection header received
0810 19:19:41.721230 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:41.721235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:41.721297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:41.721403 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:41.723334 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.724109 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.726153 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.726586 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.726676 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:41.730279 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:41.730284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:41.730287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:41.730318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:41.732710 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:41.733396 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:41.733400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:41.733402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:41.733571 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:41.733990 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:41.733993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:41.733995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:41.734255 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:41.734388 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:41.735064 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 19:19:41.735288 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260810 19:19:41.736598 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:41.717891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:41155 (local address 127.7.43.2:58277)
0810 19:19:41.718736 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:41.718774 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:41.720955 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:41.721680 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:41.721690 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:41.722390 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:41.723166 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:41.723192 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.724256 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.724261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:41.726033 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:41.726047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:41.726792 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:41.726799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:41.727060 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:41.727695 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:41.727722 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:41.730063 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:41.732850 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:41.732859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:41.732876 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:41.733273 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:41.733671 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:41.733676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:41.733680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:41.733867 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:41.734377 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:41.734385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:41.734712 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 19:19:41.734938 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 19:19:41.735467 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":291,"threads_started":1}
I20260810 19:19:41.738056 13894 heartbeater.cc:347] Connected to a master server at 127.7.43.62:41155
I20260810 19:19:41.738350 13894 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:41.739073 13894 heartbeater.cc:511] Master 127.7.43.62:41155 requested a full tablet report, sending...
I20260810 19:19:41.740547 13353 ts_manager.cc:194] Registered new tserver with Master: a1fcea8222a5432eaeaba9cc19bd43a5 (127.7.43.2:45433)
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
I20260810 19:19:41.741443 13353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:58277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:41.770112 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:41155
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:42.095149 13906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:42.095429 13906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:42.095521 13906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:42.102722 13906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:42.102834 13906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:42.102977 13906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:42.112908 13906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:42.114313 13906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:42.115630 13906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:42.129045 13914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:42.129065 13918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:42.129264 13913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:42.129966 13906 server_base.cc:1034] running on GCE node
I20260810 19:19:42.130631 13906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:42.131635 13906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:42.132831 13906 hybrid_clock.cc:648] HybridClock initialized: now 1786389582132763 us; error 53 us; skew 500 ppm
I20260810 19:19:42.133138 13906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:42.136147 13906 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:42.137673 13906 webserver.cc:460] Webserver started at http://127.7.43.3:46327/ using document root <none> and password file <none>
I20260810 19:19:42.138355 13906 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:42.138478 13906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:42.138782 13906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:42.141824 13906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "42386fb5d876487287b0f1e09b560c90"
format_stamp: "Formatted at 2026-08-10 19:19:42 on dist-test-slave-7lbf"
server_key: "a8bcbb561a3e64bc9a8fd8ebc52c8196"
server_key_iv: "be8625c48c30f740e134653d6a352934"
server_key_version: "encryptionkey@0"
I20260810 19:19:42.142586 13906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "42386fb5d876487287b0f1e09b560c90"
format_stamp: "Formatted at 2026-08-10 19:19:42 on dist-test-slave-7lbf"
server_key: "a8bcbb561a3e64bc9a8fd8ebc52c8196"
server_key_iv: "be8625c48c30f740e134653d6a352934"
server_key_version: "encryptionkey@0"
I20260810 19:19:42.147550 13906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:42.151075 13928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:42.152391 13906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:19:42.152534 13906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "42386fb5d876487287b0f1e09b560c90"
format_stamp: "Formatted at 2026-08-10 19:19:42 on dist-test-slave-7lbf"
server_key: "a8bcbb561a3e64bc9a8fd8ebc52c8196"
server_key_iv: "be8625c48c30f740e134653d6a352934"
server_key_version: "encryptionkey@0"
I20260810 19:19:42.152720 13906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:42.178066 13906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:42.181938 13906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:42.182224 13906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:42.183180 13906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:42.184504 13906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:42.184576 13906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:42.184689 13906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:42.184742 13906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:42.205678 13671 heartbeater.cc:503] Master 127.7.43.62:41155 was elected leader, sending a full tablet report...
I20260810 19:19:42.215332 13906 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:35667
I20260810 19:19:42.215377 14105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:35667 every 8 connection(s)
I20260810 19:19:42.216605 13906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:42.222657 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13906
I20260810 19:19:42.222813 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:42.223242 7340 external_mini_cluster.cc:1428] Setting key 8296917c30144e96b0a5f2c1ef06abbc
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:42.236169 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.219414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:42779 (local address 127.7.43.62:41155)
0810 19:19:42.219608 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:42.219612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:42.220752 (+ 1140us) server_negotiation.cc:407] Connection header received
0810 19:19:42.222674 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:42.222679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:42.222798 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:42.222896 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:42.224670 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.225680 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.227386 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.227782 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.227875 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.231066 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:42.231071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:42.231074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:42.231107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:42.233673 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.234305 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.234308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.234310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.234423 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.234753 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.234755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.234757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.235025 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:42.235139 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:42.235744 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 19:19:42.235918 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 19:19:42.237280 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.219851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:41155 (local address 127.7.43.3:42779)
0810 19:19:42.220583 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:42.220629 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:42.222379 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:42.223166 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:42.223174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:42.223796 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:42.224478 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:42.224504 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.225817 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.225822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:42.227256 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:42.227270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.227951 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.227956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:42.228146 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.228715 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:42.228737 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:42.230862 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:42.233789 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:42.233796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:42.233810 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:42.234218 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:42.234503 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:42.234506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:42.234508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:42.234679 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:42.235119 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:42.235124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:42.235434 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 19:19:42.235678 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 19:19:42.236052 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":282,"threads_started":1}
I20260810 19:19:42.238760 14106 heartbeater.cc:347] Connected to a master server at 127.7.43.62:41155
I20260810 19:19:42.239092 14106 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:42.239794 14106 heartbeater.cc:511] Master 127.7.43.62:41155 requested a full tablet report, sending...
I20260810 19:19:42.241137 13353 ts_manager.cc:194] Registered new tserver with Master: 42386fb5d876487287b0f1e09b560c90 (127.7.43.3:35667)
I20260810 19:19:42.241963 13353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:42779
I20260810 19:19:42.251286 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:42.307034 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.291972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.7.43.1:46685)
0810 19:19:42.292629 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:42.292636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:42.292663 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:19:42.292727 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:42.292734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:42.292954 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:42.293084 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:42.294467 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.295467 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.296671 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.297232 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.297327 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.301438 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:42.301449 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:42.301464 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:42.301496 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:42.304261 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.304889 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.304897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.304899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.305025 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.305381 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.305384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.305386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.305820 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:42.306126 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:42.306417 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 19:19:42.306687 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":329,"threads_started":1}
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:42.324921 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.311726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.7.43.62:41155)
0810 19:19:42.311944 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:42.311950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:42.312105 (+ 155us) server_negotiation.cc:407] Connection header received
0810 19:19:42.312267 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:42.312272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:42.312341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:42.312420 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:42.313620 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.314365 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.315573 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.315987 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.316095 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.319845 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:42.319850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:42.319853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:42.319887 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:42.322367 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.323012 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.323018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.323021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.323155 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.323505 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.323510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.323514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.323824 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:42.324073 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:42.324345 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 19:19:42.324593 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 19:19:42.325469 13353 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912
I20260810 19:19:42.326442 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13453
I20260810 19:19:42.337766 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.327250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.7.43.1:46685)
0810 19:19:42.327401 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:42.327406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:42.327705 (+ 299us) server_negotiation.cc:407] Connection header received
0810 19:19:42.327823 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:42.327826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:42.327917 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:42.327996 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:42.329061 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.329790 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.330947 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.331340 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.331441 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.333434 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:42.333439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:42.333442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:42.333473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:42.335768 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.336341 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.336344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.336346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.336470 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.336732 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.336735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.336737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.336963 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:42.337129 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:42.337338 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 19:19:42.337508 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:19:42.338332 13664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:42.422360 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13453
I20260810 19:19:42.446993 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13680
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:42.462808 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.447914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.7.43.2:45433)
0810 19:19:42.448450 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:42.448458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:42.448483 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:19:42.448544 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:42.448550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:42.448760 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:42.448902 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:42.450218 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.451322 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.452551 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.453100 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.453214 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.457296 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:42.457307 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:42.457322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:42.457356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:42.459945 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.460599 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.460607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.460609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.460731 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.461115 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.461118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.461120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.461486 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:42.461903 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:42.462216 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 19:19:42.462480 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":263,"threads_started":1}
I20260810 19:19:42.464084 13886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:42.545260 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13680
I20260810 19:19:42.567855 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13906
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[13279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:42.583212 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:42.568868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.7.43.3:35667)
0810 19:19:42.569475 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:42.569481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:42.569510 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:19:42.569609 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:42.569615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:42.569827 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:42.569959 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:42.571227 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.572250 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.573500 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:42.574141 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:42.574241 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:42.578314 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:42.578325 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:42.578341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:42.578372 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:42.580834 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.581413 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.581420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.581422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.581657 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:42.581953 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:42.581956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:42.581958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:42.582294 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:42.582513 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:42.582724 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 19:19:42.582925 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":300,"threads_started":1}
I20260810 19:19:42.584482 14100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:42.666177 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13906
I20260810 19:19:42.685796 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13299
I20260810 19:19:42.686995 13405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:42.790252 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13299
2026-08-10T19:19:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4138 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[14149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[14149](info): set up 2 sockets
Aug 10 19:19:42 dist-test-slave-7lbf krb5kdc[14149](info): commencing operation
krb5kdc: starting...
W20260810 19:19:44.022514 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:44 dist-test-slave-7lbf krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:44.268940 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42341
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:44.611413 14172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:44.611769 14172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:44.611922 14172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:44.619230 14172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:44.619341 14172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:44.619401 14172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:44.619464 14172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:44.619520 14172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:44.629537 14172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:42341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:44.630859 14172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:44.632229 14172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:44.643869 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:44.643849 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:44.643826 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:44.645045 14172 server_base.cc:1034] running on GCE node
I20260810 19:19:44.645817 14172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:44.647131 14172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:44.648337 14172 hybrid_clock.cc:648] HybridClock initialized: now 1786389584648293 us; error 31 us; skew 500 ppm
I20260810 19:19:44.648651 14172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:44 dist-test-slave-7lbf krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389584, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:44.651681 14172 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:44.652824 14172 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:44.652916 14172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:44.653189 14172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:44.656241 14172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "b706ee5467c748419b64b1bf29d2203c"
format_stamp: "Formatted at 2026-08-10 19:19:44 on dist-test-slave-7lbf"
server_key: "e187d18062a10d1b1e7f74b2525b86e8"
server_key_iv: "5c7fcb1109a0875c4290e401e93d6412"
server_key_version: "encryptionkey@0"
I20260810 19:19:44.656998 14172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "b706ee5467c748419b64b1bf29d2203c"
format_stamp: "Formatted at 2026-08-10 19:19:44 on dist-test-slave-7lbf"
server_key: "e187d18062a10d1b1e7f74b2525b86e8"
server_key_iv: "5c7fcb1109a0875c4290e401e93d6412"
server_key_version: "encryptionkey@0"
I20260810 19:19:44.662077 14172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:19:44.665546 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:44.666851 14172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 19:19:44.667014 14172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "b706ee5467c748419b64b1bf29d2203c"
format_stamp: "Formatted at 2026-08-10 19:19:44 on dist-test-slave-7lbf"
server_key: "e187d18062a10d1b1e7f74b2525b86e8"
server_key_iv: "5c7fcb1109a0875c4290e401e93d6412"
server_key_version: "encryptionkey@0"
I20260810 19:19:44.667220 14172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:44.692204 14172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:44.698053 14172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:44.700351 14172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:44.719493 14172 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:42341
I20260810 19:19:44.719463 14277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:42341 every 8 connection(s)
I20260810 19:19:44.720402 14172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:44.724057 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:44.724452 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 14172
I20260810 19:19:44.724630 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:44.725287 7340 external_mini_cluster.cc:1428] Setting key cbadfbaa488b273134555e987871acc2
I20260810 19:19:44.737744 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c: Bootstrap starting.
Aug 10 19:19:44 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:44.741732 14279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:44.742861 14279 log.cc:824] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:44.746090 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c: No bootstrap required, opened a new log
I20260810 19:19:44.749861 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:44.727263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.7.43.62:42341)
0810 19:19:44.728019 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:44.728031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:44.728077 (+ 46us) server_negotiation.cc:407] Connection header received
0810 19:19:44.730011 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:44.730035 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:44.730453 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:44.730806 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:44.732708 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:44.734140 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:44.735987 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:44.736705 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:44.736848 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:44.741651 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:44.741660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:44.741675 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:44.741721 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:44.745191 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:44.745922 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:44.745939 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:44.745947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:44.746167 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:44.746589 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:44.746594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:44.746597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:44.747316 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:44.747615 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:44.747985 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 19:19:44.748436 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
I20260810 19:19:44.755282 14279 raft_consensus.cc:348] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b706ee5467c748419b64b1bf29d2203c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42341 } }
I20260810 19:19:44.755604 14279 raft_consensus.cc:374] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:44.755682 14279 raft_consensus.cc:729] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706ee5467c748419b64b1bf29d2203c, State: Initialized, Role: FOLLOWER
I20260810 19:19:44.756188 14279 consensus_queue.cc:260] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b706ee5467c748419b64b1bf29d2203c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42341 } }
I20260810 19:19:44.756387 14279 raft_consensus.cc:388] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:44.756492 14279 raft_consensus.cc:482] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:44.756613 14279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:44.758873 14279 raft_consensus.cc:504] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b706ee5467c748419b64b1bf29d2203c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42341 } }
I20260810 19:19:44.759317 14279 leader_election.cc:302] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b706ee5467c748419b64b1bf29d2203c; no voters:
I20260810 19:19:44.759831 14279 leader_election.cc:288] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:44.759963 14287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:44.761273 14287 raft_consensus.cc:686] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [term 1 LEADER]: Becoming Leader. State: Replica: b706ee5467c748419b64b1bf29d2203c, State: Running, Role: LEADER
I20260810 19:19:44.761837 14287 consensus_queue.cc:237] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b706ee5467c748419b64b1bf29d2203c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42341 } }
I20260810 19:19:44.762007 14279 sys_catalog.cc:564] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:44.764341 14289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b706ee5467c748419b64b1bf29d2203c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b706ee5467c748419b64b1bf29d2203c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42341 } } }
I20260810 19:19:44.764681 14289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:44.765946 14290 sys_catalog.cc:455] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b706ee5467c748419b64b1bf29d2203c. Latest consensus state: current_term: 1 leader_uuid: "b706ee5467c748419b64b1bf29d2203c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b706ee5467c748419b64b1bf29d2203c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42341 } } }
I20260810 19:19:44.766500 14290 sys_catalog.cc:458] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:44.767735 14295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:44.772073 14295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:44.779886 14295 catalog_manager.cc:1347] Generated new cluster ID: d776153c368346fb89247eace85f2a19
I20260810 19:19:44.780002 14295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:44.793121 14295 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:44.795109 14295 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:44.809115 14295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c: Generated new TSK 0
I20260810 19:19:44.810298 14295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:44.873168 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42341
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:45.218693 14327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:45.218992 14327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:45.219090 14327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:45.226245 14327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:45.226362 14327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:45.226487 14327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:45.236583 14327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:45.237974 14327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:45.239253 14327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:45.253237 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:45.253334 14327 server_base.cc:1034] running on GCE node
W20260810 19:19:45.253441 14342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:45.253273 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:45.254252 14327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:45.255139 14327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:45.256340 14327 hybrid_clock.cc:648] HybridClock initialized: now 1786389585256288 us; error 36 us; skew 500 ppm
I20260810 19:19:45.256659 14327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:45 dist-test-slave-7lbf krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389585, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:45.259759 14327 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:45.260874 14327 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:45.260968 14327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:45.261283 14327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:45.264492 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "a31ef1e82c3b486489d272f89f573d31"
format_stamp: "Formatted at 2026-08-10 19:19:45 on dist-test-slave-7lbf"
server_key: "3f6e60bd9dff3fa6398b1a7623dcbe5c"
server_key_iv: "1102002707cc44f17d68afd2462e20a5"
server_key_version: "encryptionkey@0"
I20260810 19:19:45.265324 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "a31ef1e82c3b486489d272f89f573d31"
format_stamp: "Formatted at 2026-08-10 19:19:45 on dist-test-slave-7lbf"
server_key: "3f6e60bd9dff3fa6398b1a7623dcbe5c"
server_key_iv: "1102002707cc44f17d68afd2462e20a5"
server_key_version: "encryptionkey@0"
I20260810 19:19:45.270871 14327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:19:45.274271 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:45.275530 14327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:45.275677 14327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "a31ef1e82c3b486489d272f89f573d31"
format_stamp: "Formatted at 2026-08-10 19:19:45 on dist-test-slave-7lbf"
server_key: "3f6e60bd9dff3fa6398b1a7623dcbe5c"
server_key_iv: "1102002707cc44f17d68afd2462e20a5"
server_key_version: "encryptionkey@0"
I20260810 19:19:45.275869 14327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:45.313068 14327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:45.317133 14327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:45.317386 14327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:45.318401 14327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:45.319729 14327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:45.319823 14327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:45.319928 14327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:45.320008 14327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:45.350951 14327 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34213
I20260810 19:19:45.351011 14541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34213 every 8 connection(s)
I20260810 19:19:45.351814 14327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:45.360146 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 14327
I20260810 19:19:45.360353 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:45.361172 7340 external_mini_cluster.cc:1428] Setting key 15444a97b7d5158c13a1305c09f69476
Aug 10 19:19:45 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389585, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:45.372401 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:45.354187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:41273 (local address 127.7.43.62:42341)
0810 19:19:45.354653 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:45.354659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:45.355652 (+ 993us) server_negotiation.cc:407] Connection header received
0810 19:19:45.357482 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:45.357487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:45.357603 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:45.357694 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:45.359086 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.359773 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.362054 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.362591 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.362713 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:45.366402 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:45.366406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:45.366410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:45.366440 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:45.369337 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:45.370059 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:45.370064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:45.370067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:45.370217 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:45.370623 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:45.370628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:45.370631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:45.371004 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:45.371134 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:45.371861 (+ 727us) server_negotiation.cc:299] Negotiation successful
0810 19:19:45.372077 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260810 19:19:45.373306 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:45.354749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42341 (local address 127.7.43.1:41273)
0810 19:19:45.355485 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:45.355525 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:45.357217 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:45.357884 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:45.357892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:45.358382 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:45.358953 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:45.358970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.359951 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.359958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:45.361898 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:45.361910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.362783 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.362789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:45.363046 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:45.363720 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:45.363750 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:45.366189 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:45.369474 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:45.369481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:45.369494 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:45.369952 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:45.370306 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:45.370309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:45.370311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:45.370505 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:45.371109 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:45.371115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:45.371467 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 19:19:45.371703 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 19:19:45.372177 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":287,"threads_started":1}
I20260810 19:19:45.374950 14542 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42341
I20260810 19:19:45.375242 14542 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:45.375895 14542 heartbeater.cc:511] Master 127.7.43.62:42341 requested a full tablet report, sending...
I20260810 19:19:45.378206 14219 ts_manager.cc:194] Registered new tserver with Master: a31ef1e82c3b486489d272f89f573d31 (127.7.43.1:34213)
I20260810 19:19:45.380192 14219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:41273
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:45.417367 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42341
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:45.741683 14553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:45.741966 14553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:45.742082 14553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:45.749202 14553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:45.749333 14553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:45.749547 14553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:45.759789 14553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:45.761162 14553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:45.762612 14553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:45.776418 14566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:45.776366 14563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:45.776348 14562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:45.777221 14553 server_base.cc:1034] running on GCE node
I20260810 19:19:45.778010 14553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:45.779137 14553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:45.780393 14553 hybrid_clock.cc:648] HybridClock initialized: now 1786389585780298 us; error 75 us; skew 500 ppm
I20260810 19:19:45.780733 14553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:45 dist-test-slave-7lbf krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389585, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:45.786965 14553 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:45.788184 14553 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:45.788301 14553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:45.788618 14553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:45.791672 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "9a8e961a6bc842e190521a8e2295952d"
format_stamp: "Formatted at 2026-08-10 19:19:45 on dist-test-slave-7lbf"
server_key: "d9badfb2bab866c73865211e4e069e94"
server_key_iv: "7473462c13a72594262c5b45649b3516"
server_key_version: "encryptionkey@0"
I20260810 19:19:45.792492 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "9a8e961a6bc842e190521a8e2295952d"
format_stamp: "Formatted at 2026-08-10 19:19:45 on dist-test-slave-7lbf"
server_key: "d9badfb2bab866c73865211e4e069e94"
server_key_iv: "7473462c13a72594262c5b45649b3516"
server_key_version: "encryptionkey@0"
I20260810 19:19:45.797555 14553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:19:45.800930 14581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:45.802238 14553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:19:45.802393 14553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "9a8e961a6bc842e190521a8e2295952d"
format_stamp: "Formatted at 2026-08-10 19:19:45 on dist-test-slave-7lbf"
server_key: "d9badfb2bab866c73865211e4e069e94"
server_key_iv: "7473462c13a72594262c5b45649b3516"
server_key_version: "encryptionkey@0"
I20260810 19:19:45.802596 14553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:45.833227 14553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:45.837085 14553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:45.837330 14553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:45.838297 14553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:45.839591 14553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:45.839686 14553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:45.839790 14553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:45.839864 14553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:45.879179 14553 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:37435
I20260810 19:19:45.879200 14773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:37435 every 8 connection(s)
I20260810 19:19:45.880159 14553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:45.889038 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 14553
I20260810 19:19:45.889223 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:45.889854 7340 external_mini_cluster.cc:1428] Setting key f390f59890924ced124f0b34642cb4be
Aug 10 19:19:45 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389585, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:45.901324 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:45.882878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:58661 (local address 127.7.43.62:42341)
0810 19:19:45.883382 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:45.883389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:45.884462 (+ 1073us) server_negotiation.cc:407] Connection header received
0810 19:19:45.886347 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:45.886351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:45.886423 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:45.886539 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:45.888029 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.888739 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.891015 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.891681 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.891806 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:45.895810 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:45.895815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:45.895819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:45.895852 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:45.898329 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:45.899168 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:45.899172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:45.899174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:45.899296 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:45.899717 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:45.899720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:45.899722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:45.899980 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:45.900096 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:45.900890 (+ 794us) server_negotiation.cc:299] Negotiation successful
0810 19:19:45.901054 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":221,"threads_started":1}
I20260810 19:19:45.902524 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:45.883530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42341 (local address 127.7.43.2:58661)
0810 19:19:45.884289 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:45.884329 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:45.886057 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:45.886740 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:45.886749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:45.887283 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:45.887880 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:45.887903 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.888901 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.888906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:45.890848 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:45.890863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:45.891901 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:45.891909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:45.892150 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:45.892854 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:45.892884 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:45.895558 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:45.898471 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:45.898480 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:45.898499 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:45.899064 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:45.899388 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:45.899392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:45.899395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:45.899624 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:45.900091 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:45.900098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:45.900528 (+ 430us) client_negotiation.cc:769] Sending connection context
0810 19:19:45.900827 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 19:19:45.901363 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":318,"threads_started":1}
I20260810 19:19:45.904135 14774 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42341
I20260810 19:19:45.904434 14774 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:45.905069 14774 heartbeater.cc:511] Master 127.7.43.62:42341 requested a full tablet report, sending...
I20260810 19:19:45.906697 14219 ts_manager.cc:194] Registered new tserver with Master: 9a8e961a6bc842e190521a8e2295952d (127.7.43.2:37435)
I20260810 19:19:45.907747 14219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:58661
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:45.944437 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42341
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:46.269968 14784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:46.270241 14784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:46.270361 14784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:46.277714 14784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:46.277818 14784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:46.277968 14784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:46.287643 14784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:46.288892 14784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:46.290227 14784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:46.302361 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:46.302389 14800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:46.302330 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:46.303082 14784 server_base.cc:1034] running on GCE node
I20260810 19:19:46.303653 14784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:46.304576 14784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:46.305781 14784 hybrid_clock.cc:648] HybridClock initialized: now 1786389586305726 us; error 45 us; skew 500 ppm
I20260810 19:19:46.306092 14784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:46 dist-test-slave-7lbf krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389586, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:46.309072 14784 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:46.310178 14784 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:46.310283 14784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:46.310606 14784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:46.313438 14784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "7c9ac333c2ff4af8b8de3d584869d57d"
format_stamp: "Formatted at 2026-08-10 19:19:46 on dist-test-slave-7lbf"
server_key: "9b67e915aafb5a256c43b757b61cc3aa"
server_key_iv: "31cfc5f25a580ef54e3f1ed1b0a5b41f"
server_key_version: "encryptionkey@0"
I20260810 19:19:46.314278 14784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "7c9ac333c2ff4af8b8de3d584869d57d"
format_stamp: "Formatted at 2026-08-10 19:19:46 on dist-test-slave-7lbf"
server_key: "9b67e915aafb5a256c43b757b61cc3aa"
server_key_iv: "31cfc5f25a580ef54e3f1ed1b0a5b41f"
server_key_version: "encryptionkey@0"
I20260810 19:19:46.319135 14784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:46.322520 14806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:46.323846 14784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:46.324009 14784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "7c9ac333c2ff4af8b8de3d584869d57d"
format_stamp: "Formatted at 2026-08-10 19:19:46 on dist-test-slave-7lbf"
server_key: "9b67e915aafb5a256c43b757b61cc3aa"
server_key_iv: "31cfc5f25a580ef54e3f1ed1b0a5b41f"
server_key_version: "encryptionkey@0"
I20260810 19:19:46.324208 14784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:46.338215 14784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:46.341930 14784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:46.342191 14784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:46.343119 14784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:46.344409 14784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:46.344507 14784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:46.344610 14784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:46.344677 14784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:46.375142 14784 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:42863
I20260810 19:19:46.375193 14990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:42863 every 8 connection(s)
I20260810 19:19:46.375983 14784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:46.383373 14542 heartbeater.cc:503] Master 127.7.43.62:42341 was elected leader, sending a full tablet report...
I20260810 19:19:46.386086 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 14784
I20260810 19:19:46.386240 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:46.386695 7340 external_mini_cluster.cc:1428] Setting key b14dc33f80d1700f46699d7d9c36e980
Aug 10 19:19:46 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389586, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:46.395396 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.378419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:53667 (local address 127.7.43.62:42341)
0810 19:19:46.378605 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.378610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.379866 (+ 1256us) server_negotiation.cc:407] Connection header received
0810 19:19:46.381835 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.381841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.381933 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.382033 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:46.383676 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.384592 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.386310 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.386711 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.386820 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.390375 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:46.390380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:46.390383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:46.390415 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:46.392762 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.393427 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.393431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.393433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.393596 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.393956 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.393959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.393961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.394235 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:46.394344 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.394989 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.395166 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:19:46.396339 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.378973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42341 (local address 127.7.43.3:53667)
0810 19:19:46.379698 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:46.379744 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:46.381508 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:46.382305 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:46.382314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:46.382866 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:46.383472 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.383500 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.384742 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.384746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.386165 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.386181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.386935 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.386942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.387160 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.387807 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:46.387838 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:46.390174 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:46.392880 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:46.392887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:46.392901 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:46.393332 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:46.393683 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:46.393686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:46.393689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:46.393867 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:46.394322 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:46.394328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:46.394660 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 19:19:46.394889 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 19:19:46.395294 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":270,"threads_started":1}
I20260810 19:19:46.397693 14992 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42341
I20260810 19:19:46.397984 14992 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:46.398619 14992 heartbeater.cc:511] Master 127.7.43.62:42341 requested a full tablet report, sending...
I20260810 19:19:46.400005 14219 ts_manager.cc:194] Registered new tserver with Master: 7c9ac333c2ff4af8b8de3d584869d57d (127.7.43.3:42863)
I20260810 19:19:46.400888 14219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:53667
I20260810 19:19:46.402108 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:19:46.420105 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.406872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.7.43.62:42341)
0810 19:19:46.407026 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.407032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.407537 (+ 505us) server_negotiation.cc:407] Connection header received
0810 19:19:46.407672 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.407676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.407734 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.407813 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:46.409203 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.410058 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.411880 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.412442 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.412559 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.414994 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:46.414999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:46.415002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:46.415035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:46.417484 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.418152 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.418158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.418161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.418282 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.418641 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.418646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.418649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.418990 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:46.419227 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.419560 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.419775 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 19:19:46.423950 14219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:46.430965 14219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:46.490460 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.444047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34213 (local address 127.0.0.1:57224)
0810 19:19:46.482310 (+ 38263us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:46.482333 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:46.482474 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:46.483358 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:46.483367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:46.483400 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:46.483695 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.483705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.485613 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.485619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.487563 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.487579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.489428 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.489434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.489654 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.489714 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 19:19:46.489841 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 19:19:46.490047 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38152,"mutex_wait_us":54,"thread_start_us":36051,"threads_started":1}
I20260810 19:19:46.491273 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.444750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.7.43.1:34213)
0810 19:19:46.477824 (+ 33074us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.477836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.482474 (+ 4638us) server_negotiation.cc:407] Connection header received
0810 19:19:46.482650 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.482659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.482982 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.483214 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:46.483865 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.485429 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.487734 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.489231 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.489350 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.489621 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.490744 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.490918 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32816,"thread_start_us":33105,"threads_started":1}
I20260810 19:19:46.491882 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.443371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:37435 (local address 127.0.0.1:49186)
0810 19:19:46.479414 (+ 36043us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:46.479496 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:46.479756 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:46.485216 (+ 5460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:46.485221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:46.485241 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:46.485651 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.485663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.487316 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.487319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.489565 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.489581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.491119 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.491126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.491250 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.491273 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:19:46.491343 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 19:19:46.491471 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35687,"mutex_wait_us":26,"thread_start_us":39188,"threads_started":1}
I20260810 19:19:46.492435 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.443631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.7.43.2:37435)
0810 19:19:46.484277 (+ 40646us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.484286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.484311 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:19:46.484407 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.484414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.484636 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.485708 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:46.485922 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.487167 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.489718 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.490664 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.490774 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.490966 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.491991 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.492147 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40491,"thread_start_us":40341,"threads_started":1}
I20260810 19:19:46.492877 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.444360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.7.43.3:42863)
0810 19:19:46.483390 (+ 39030us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.483400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.483448 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:19:46.483564 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.483574 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.483928 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.484842 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:46.485104 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.486638 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.489502 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.490959 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.491081 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.491294 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.491815 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.491990 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38826,"thread_start_us":38637,"threads_started":1}
I20260810 19:19:46.493005 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.443847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:42863 (local address 127.0.0.1:38070)
0810 19:19:46.481721 (+ 37874us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:46.481747 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:46.482747 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:46.484150 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:46.484155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:46.484181 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:46.484613 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.484627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.486785 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.486791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.489292 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.489306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.491457 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.491463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.491633 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.491657 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:19:46.491740 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:19:46.491872 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37733,"mutex_wait_us":67,"thread_start_us":37575,"threads_started":1}
I20260810 19:19:46.495440 14439 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3286d4214c419b80c87fabad5eb4b2 (DEFAULT_TABLE table=test-table [id=b155e38f90b644ccb4d8dece0c29fcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:46.496320 14893 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3286d4214c419b80c87fabad5eb4b2 (DEFAULT_TABLE table=test-table [id=b155e38f90b644ccb4d8dece0c29fcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:46.497061 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3286d4214c419b80c87fabad5eb4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:46.496730 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3286d4214c419b80c87fabad5eb4b2 (DEFAULT_TABLE table=test-table [id=b155e38f90b644ccb4d8dece0c29fcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:46.498081 14893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3286d4214c419b80c87fabad5eb4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:46.498487 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3286d4214c419b80c87fabad5eb4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:46.509248 15022 tablet_bootstrap.cc:492] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Bootstrap starting.
I20260810 19:19:46.512964 15022 tablet_bootstrap.cc:654] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:46.513872 15027 tablet_bootstrap.cc:492] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Bootstrap starting.
I20260810 19:19:46.514317 15022 log.cc:824] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:46.515807 15025 tablet_bootstrap.cc:492] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Bootstrap starting.
I20260810 19:19:46.517603 15027 tablet_bootstrap.cc:654] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:46.517843 15022 tablet_bootstrap.cc:492] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: No bootstrap required, opened a new log
I20260810 19:19:46.518092 15022 ts_tablet_manager.cc:1397] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:19:46.518859 15027 log.cc:824] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:46.519189 15025 tablet_bootstrap.cc:654] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:46.520468 15025 log.cc:824] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:46.522851 15027 tablet_bootstrap.cc:492] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: No bootstrap required, opened a new log
I20260810 19:19:46.523104 15027 ts_tablet_manager.cc:1397] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260810 19:19:46.523541 15025 tablet_bootstrap.cc:492] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: No bootstrap required, opened a new log
I20260810 19:19:46.523803 15025 ts_tablet_manager.cc:1397] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:19:46.527642 15022 raft_consensus.cc:348] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.528105 15022 raft_consensus.cc:374] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:46.528215 15022 raft_consensus.cc:729] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9ac333c2ff4af8b8de3d584869d57d, State: Initialized, Role: FOLLOWER
I20260810 19:19:46.528863 15022 consensus_queue.cc:260] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.530154 14992 heartbeater.cc:503] Master 127.7.43.62:42341 was elected leader, sending a full tablet report...
I20260810 19:19:46.530793 15022 ts_tablet_manager.cc:1428] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260810 19:19:46.531170 15027 raft_consensus.cc:348] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.531599 15027 raft_consensus.cc:374] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:46.531714 15027 raft_consensus.cc:729] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8e961a6bc842e190521a8e2295952d, State: Initialized, Role: FOLLOWER
I20260810 19:19:46.532339 15027 consensus_queue.cc:260] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.533353 14774 heartbeater.cc:503] Master 127.7.43.62:42341 was elected leader, sending a full tablet report...
I20260810 19:19:46.534013 15027 ts_tablet_manager.cc:1428] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 19:19:46.534997 15025 raft_consensus.cc:348] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.535326 15025 raft_consensus.cc:374] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:46.535430 15025 raft_consensus.cc:729] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31ef1e82c3b486489d272f89f573d31, State: Initialized, Role: FOLLOWER
I20260810 19:19:46.536016 15025 consensus_queue.cc:260] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.537659 15025 ts_tablet_manager.cc:1428] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260810 19:19:46.599648 15038 raft_consensus.cc:482] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:46.599961 15038 raft_consensus.cc:504] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.601732 15038 leader_election.cc:288] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a31ef1e82c3b486489d272f89f573d31 (127.7.43.1:34213), 7c9ac333c2ff4af8b8de3d584869d57d (127.7.43.3:42863)
W20260810 19:19:46.603996 14543 tablet.cc:2369] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:46.608911 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.602280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:49595 (local address 127.7.43.3:42863)
0810 19:19:46.602512 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.602518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.602823 (+ 305us) server_negotiation.cc:407] Connection header received
0810 19:19:46.603137 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.603143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.603218 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.603288 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:46.604271 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.605093 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.607035 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.607998 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.608090 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.608164 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.608543 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.608667 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:19:46.608935 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.602004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:42863 (local address 127.7.43.2:49595)
0810 19:19:46.602695 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:46.602719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:46.602961 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:46.603504 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:46.603509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:46.603534 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:46.604036 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.604050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.605205 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.605211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.606883 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.606900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.608203 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.608210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.608378 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.608413 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:19:46.608511 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:19:46.608648 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":315,"threads_started":1}
I20260810 19:19:46.609350 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.602242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:48009 (local address 127.7.43.1:34213)
0810 19:19:46.602475 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.602482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.603067 (+ 585us) server_negotiation.cc:407] Connection header received
0810 19:19:46.603196 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.603201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.603283 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.603379 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:19:46.604463 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.605556 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.607507 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.608382 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.608511 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.608607 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.609004 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.609130 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:19:46.609452 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.602004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34213 (local address 127.7.43.2:48009)
0810 19:19:46.602951 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:46.602968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:46.603085 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:46.603846 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:46.603851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:46.603873 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:19:46.604259 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.604270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.605698 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.605702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.607356 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:46.607372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.608638 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.608644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:46.608828 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.608859 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:19:46.608963 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 19:19:46.609083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":199,"thread_start_us":434,"threads_started":1}
I20260810 19:19:46.610148 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3286d4214c419b80c87fabad5eb4b2" candidate_uuid: "9a8e961a6bc842e190521a8e2295952d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" is_pre_election: true
I20260810 19:19:46.610459 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3286d4214c419b80c87fabad5eb4b2" candidate_uuid: "9a8e961a6bc842e190521a8e2295952d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31ef1e82c3b486489d272f89f573d31" is_pre_election: true
I20260810 19:19:46.610661 14922 raft_consensus.cc:2454] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a8e961a6bc842e190521a8e2295952d in term 0.
I20260810 19:19:46.610941 14476 raft_consensus.cc:2454] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a8e961a6bc842e190521a8e2295952d in term 0.
I20260810 19:19:46.611323 14584 leader_election.cc:302] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c9ac333c2ff4af8b8de3d584869d57d, 9a8e961a6bc842e190521a8e2295952d; no voters:
I20260810 19:19:46.611747 15038 raft_consensus.cc:2781] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:46.611901 15038 raft_consensus.cc:482] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:46.612003 15038 raft_consensus.cc:3037] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:46.614207 15038 raft_consensus.cc:504] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.614782 15038 leader_election.cc:288] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [CANDIDATE]: Term 1 election: Requested vote from peers a31ef1e82c3b486489d272f89f573d31 (127.7.43.1:34213), 7c9ac333c2ff4af8b8de3d584869d57d (127.7.43.3:42863)
I20260810 19:19:46.615181 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3286d4214c419b80c87fabad5eb4b2" candidate_uuid: "9a8e961a6bc842e190521a8e2295952d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31ef1e82c3b486489d272f89f573d31"
I20260810 19:19:46.615302 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3286d4214c419b80c87fabad5eb4b2" candidate_uuid: "9a8e961a6bc842e190521a8e2295952d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9ac333c2ff4af8b8de3d584869d57d"
I20260810 19:19:46.615391 14476 raft_consensus.cc:3037] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:46.615502 14922 raft_consensus.cc:3037] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:46.618551 14922 raft_consensus.cc:2454] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8e961a6bc842e190521a8e2295952d in term 1.
I20260810 19:19:46.618561 14476 raft_consensus.cc:2454] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8e961a6bc842e190521a8e2295952d in term 1.
I20260810 19:19:46.618973 14586 leader_election.cc:302] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7c9ac333c2ff4af8b8de3d584869d57d, 9a8e961a6bc842e190521a8e2295952d, a31ef1e82c3b486489d272f89f573d31; no voters:
I20260810 19:19:46.619305 15038 raft_consensus.cc:2781] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:46.619714 15038 raft_consensus.cc:686] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 1 LEADER]: Becoming Leader. State: Replica: 9a8e961a6bc842e190521a8e2295952d, State: Running, Role: LEADER
I20260810 19:19:46.620193 15038 consensus_queue.cc:237] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } }
I20260810 19:19:46.624838 14219 catalog_manager.cc:5495] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a8e961a6bc842e190521a8e2295952d (127.7.43.2). New cstate: current_term: 1 leader_uuid: "9a8e961a6bc842e190521a8e2295952d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8e961a6bc842e190521a8e2295952d" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 } health_report { overall_health: UNKNOWN } } }
W20260810 19:19:46.627359 14993 tablet.cc:2369] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:19:46.632004 14776 tablet.cc:2369] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:46.645046 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.639887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.7.43.2:37435)
0810 19:19:46.640083 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.640088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.640106 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:19:46.640293 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.640297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.640371 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.640504 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:19:46.640871 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.641632 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.642974 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.643399 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.643509 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.643980 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:19:46.644476 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:19:46.644621 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.644711 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.644791 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:19:46.653190 14476 raft_consensus.cc:1261] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 1 FOLLOWER]: Refusing update from remote peer 9a8e961a6bc842e190521a8e2295952d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:46.653191 14922 raft_consensus.cc:1261] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 1 FOLLOWER]: Refusing update from remote peer 9a8e961a6bc842e190521a8e2295952d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:46.654145 15044 consensus_queue.cc:1035] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a31ef1e82c3b486489d272f89f573d31" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:46.654488 15038 consensus_queue.cc:1035] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9ac333c2ff4af8b8de3d584869d57d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:46.665704 15051 mvcc.cc:205] Tried to move back new op lower bound from 7317051746921353216 to 7317051746800168960. Current Snapshot: MvccSnapshot[applied={T|T < 7317051746921353216}]
I20260810 19:19:46.682412 14219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:46.682708 14219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:46.686307 14219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b706ee5467c748419b64b1bf29d2203c: Sending DeleteTablet for 3 replicas of tablet 0e3286d4214c419b80c87fabad5eb4b2
I20260810 19:19:46.687508 14439 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3286d4214c419b80c87fabad5eb4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:46 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:57224
I20260810 19:19:46.687557 14672 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3286d4214c419b80c87fabad5eb4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:46 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:49186
I20260810 19:19:46.687925 14893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3286d4214c419b80c87fabad5eb4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:46 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:38070
I20260810 19:19:46.689488 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 14327
I20260810 19:19:46.702800 15060 tablet_replica.cc:330] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: stopping tablet replica
I20260810 19:19:46.703293 15060 raft_consensus.cc:2229] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:46.703915 15062 tablet_replica.cc:330] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: stopping tablet replica
I20260810 19:19:46.704500 15062 raft_consensus.cc:2229] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:46.705103 15062 raft_consensus.cc:2258] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:46.705902 15060 raft_consensus.cc:2258] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:46.708209 15062 ts_tablet_manager.cc:1905] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:46.709075 15060 ts_tablet_manager.cc:1905] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:46.714748 15064 tablet_replica.cc:330] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: stopping tablet replica
I20260810 19:19:46.715866 15062 ts_tablet_manager.cc:1918] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:46.716080 15062 log.cc:1196] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/0e3286d4214c419b80c87fabad5eb4b2
I20260810 19:19:46.716622 15060 ts_tablet_manager.cc:1918] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:46.716673 15062 ts_tablet_manager.cc:1939] T 0e3286d4214c419b80c87fabad5eb4b2 P a31ef1e82c3b486489d272f89f573d31: Deleting consensus metadata
I20260810 19:19:46.716797 15060 log.cc:1196] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/0e3286d4214c419b80c87fabad5eb4b2
I20260810 19:19:46.717401 15060 ts_tablet_manager.cc:1939] T 0e3286d4214c419b80c87fabad5eb4b2 P 9a8e961a6bc842e190521a8e2295952d: Deleting consensus metadata
Aug 10 19:19:46 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:46.718838 14196 catalog_manager.cc:4841] TS a31ef1e82c3b486489d272f89f573d31 (127.7.43.1:34213): tablet 0e3286d4214c419b80c87fabad5eb4b2 (table test-table [id=b155e38f90b644ccb4d8dece0c29fcb1]) successfully deleted
I20260810 19:19:46.719139 14198 catalog_manager.cc:4841] TS 9a8e961a6bc842e190521a8e2295952d (127.7.43.2:37435): tablet 0e3286d4214c419b80c87fabad5eb4b2 (table test-table [id=b155e38f90b644ccb4d8dece0c29fcb1]) successfully deleted
I20260810 19:19:46.720141 15064 raft_consensus.cc:2229] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:46.720655 15064 raft_consensus.cc:2258] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:46.723026 15064 ts_tablet_manager.cc:1905] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:46.726848 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.690994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.7.43.1:34213)
0810 19:19:46.691175 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.691182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.707277 (+ 16095us) server_negotiation.cc:407] Connection header received
0810 19:19:46.707588 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.707594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.707689 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.707764 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:46.709693 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.710838 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.712679 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.713361 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.713458 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.719826 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:46.719839 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:46.719856 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:46.719894 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:46.723473 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.724291 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.724300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.724304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.724476 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.724919 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.724924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.724928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.725380 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:46.725700 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.726258 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.726501 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:19:46.728138 14533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:46.733819 15064 ts_tablet_manager.cc:1918] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:46.734014 15064 log.cc:1196] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/0e3286d4214c419b80c87fabad5eb4b2
I20260810 19:19:46.734544 15064 ts_tablet_manager.cc:1939] T 0e3286d4214c419b80c87fabad5eb4b2 P 7c9ac333c2ff4af8b8de3d584869d57d: Deleting consensus metadata
I20260810 19:19:46.736438 14195 catalog_manager.cc:4841] TS 7c9ac333c2ff4af8b8de3d584869d57d (127.7.43.3:42863): tablet 0e3286d4214c419b80c87fabad5eb4b2 (table test-table [id=b155e38f90b644ccb4d8dece0c29fcb1]) successfully deleted
I20260810 19:19:46.820194 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 14327
I20260810 19:19:46.850369 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 14553
Aug 10 19:19:46 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:46.867117 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.851310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.7.43.2:37435)
0810 19:19:46.851474 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:46.851478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:46.851496 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:19:46.851597 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:46.851604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:46.851665 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:46.851750 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:46.853316 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.854174 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.855940 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:46.856374 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:46.856473 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:46.861371 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:46.861384 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:46.861400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:46.861449 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:46.864395 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.865162 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.865168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.865170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.865317 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:46.865723 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:46.865726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:46.865728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:46.866082 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:46.866391 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:46.866635 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 19:19:46.866794 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:19:46.867884 14766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:46.978758 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 14553
I20260810 19:19:46.999119 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 14784
Aug 10 19:19:47 dist-test-slave-7lbf krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:47.013379 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:46.999885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.7.43.3:42863)
0810 19:19:47.000081 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:47.000088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:47.000108 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:19:47.000262 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:47.000267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:47.000333 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:47.000418 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:47.001725 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:47.002463 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:47.003714 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:47.004118 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:47.004227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:47.008218 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:47.008231 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:47.008247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:47.008286 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:47.010781 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:47.011394 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:47.011401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:47.011403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:47.011544 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:47.011900 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:47.011903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:47.011905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:47.012267 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:47.012540 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:47.012858 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 19:19:47.013088 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:19:47.014020 14983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:47.100656 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 14784
I20260810 19:19:47.124729 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 14172
I20260810 19:19:47.125923 14272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:47.232707 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 14172
2026-08-10T19:19:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4439 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T19:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:47Z Disabled control of system clock
I20260810 19:19:47.315382 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39759
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 19:19:47.647524 15079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:47.647835 15079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:47.647964 15079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:47.655746 15079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:47.655856 15079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:47.655897 15079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:47.655990 15079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:47.656042 15079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:47.665463 15079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:47.666805 15079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:47.668596 15079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:47.680115 15085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:47.680167 15086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:47.680212 15088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:47.681041 15079 server_base.cc:1034] running on GCE node
I20260810 19:19:47.681862 15079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:47.683207 15079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:47.684414 15079 hybrid_clock.cc:648] HybridClock initialized: now 1786389587684355 us; error 46 us; skew 500 ppm
I20260810 19:19:47.684751 15079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:47.687050 15079 webserver.cc:460] Webserver started at http://127.7.43.62:43913/ using document root <none> and password file <none>
I20260810 19:19:47.687709 15079 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:47.687811 15079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:47.688133 15079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:47.691092 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "68bde36af8324c048c0b2f9e798069e9"
format_stamp: "Formatted at 2026-08-10 19:19:47 on dist-test-slave-7lbf"
server_key: "cbf6a1971202f8940e3710e0810cd102"
server_key_iv: "9edb2d9217fba93df1d4506bfe597492"
server_key_version: "encryptionkey@0"
I20260810 19:19:47.691864 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "68bde36af8324c048c0b2f9e798069e9"
format_stamp: "Formatted at 2026-08-10 19:19:47 on dist-test-slave-7lbf"
server_key: "cbf6a1971202f8940e3710e0810cd102"
server_key_iv: "9edb2d9217fba93df1d4506bfe597492"
server_key_version: "encryptionkey@0"
I20260810 19:19:47.696964 15079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:19:47.700417 15097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:47.701752 15079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:47.701911 15079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "68bde36af8324c048c0b2f9e798069e9"
format_stamp: "Formatted at 2026-08-10 19:19:47 on dist-test-slave-7lbf"
server_key: "cbf6a1971202f8940e3710e0810cd102"
server_key_iv: "9edb2d9217fba93df1d4506bfe597492"
server_key_version: "encryptionkey@0"
I20260810 19:19:47.702119 15079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:47.707819 15079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:47.708556 15079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:47.708829 15079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:47.726070 15079 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:39759
I20260810 19:19:47.726116 15194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:39759 every 8 connection(s)
I20260810 19:19:47.727382 15079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:47.731005 15195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:47.737241 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 15079
I20260810 19:19:47.737475 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:47.738334 7340 external_mini_cluster.cc:1428] Setting key e1dc8bbd3828d2be241d3acaab26fb28
I20260810 19:19:47.741986 15195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9: Bootstrap starting.
I20260810 19:19:47.745294 15195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:47.746477 15195 log.cc:824] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:47.746742 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:47.740479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.7.43.62:39759)
0810 19:19:47.741433 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:47.741452 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:47.741493 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:19:47.743240 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:47.743265 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:47.743679 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:47.743998 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:47.745069 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:47.745076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:47.745088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:47.745102 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:47.745165 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:47.745185 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:47.745345 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:47.745500 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 19:19:47.745803 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":315,"threads_started":1}
I20260810 19:19:47.750154 15195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9: No bootstrap required, opened a new log
I20260810 19:19:47.756989 15195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bde36af8324c048c0b2f9e798069e9" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39759 } }
I20260810 19:19:47.757311 15195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:47.757404 15195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bde36af8324c048c0b2f9e798069e9, State: Initialized, Role: FOLLOWER
I20260810 19:19:47.757961 15195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bde36af8324c048c0b2f9e798069e9" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39759 } }
I20260810 19:19:47.758167 15195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:47.758325 15195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:47.758515 15195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:47.760792 15195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bde36af8324c048c0b2f9e798069e9" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39759 } }
I20260810 19:19:47.761257 15195 leader_election.cc:302] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bde36af8324c048c0b2f9e798069e9; no voters:
I20260810 19:19:47.761827 15195 leader_election.cc:288] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:47.761938 15203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:47.762584 15203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [term 1 LEADER]: Becoming Leader. State: Replica: 68bde36af8324c048c0b2f9e798069e9, State: Running, Role: LEADER
I20260810 19:19:47.763068 15203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bde36af8324c048c0b2f9e798069e9" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39759 } }
I20260810 19:19:47.763746 15195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:47.765753 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bde36af8324c048c0b2f9e798069e9. Latest consensus state: current_term: 1 leader_uuid: "68bde36af8324c048c0b2f9e798069e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bde36af8324c048c0b2f9e798069e9" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39759 } } }
I20260810 19:19:47.766153 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:47.765877 15204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bde36af8324c048c0b2f9e798069e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bde36af8324c048c0b2f9e798069e9" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39759 } } }
I20260810 19:19:47.767390 15204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:47.768839 15217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:47.773888 15217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:47.782986 15217 catalog_manager.cc:1347] Generated new cluster ID: a6d6fe7b126c403793db6da9bd20ffad
I20260810 19:19:47.783103 15217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:47.797564 15217 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:47.797993 15217 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:47.804263 15217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9: Generated new TSK 0
I20260810 19:19:47.805086 15217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:19:47.817643 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39759
--builtin_ntp_servers=127.7.43.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 19:19:48.176504 15234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:48.176782 15234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:48.176879 15234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:48.183997 15234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:48.184129 15234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:48.184262 15234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:48.193285 15234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:48.194701 15234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:48.196448 15234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:48.209383 15240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:48.209474 15243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:48.209416 15239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:48.210271 15234 server_base.cc:1034] running on GCE node
I20260810 19:19:48.210881 15234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:48.211800 15234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:48.213023 15234 hybrid_clock.cc:648] HybridClock initialized: now 1786389588212962 us; error 51 us; skew 500 ppm
I20260810 19:19:48.213330 15234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:48.215581 15234 webserver.cc:460] Webserver started at http://127.7.43.1:38117/ using document root <none> and password file <none>
I20260810 19:19:48.216254 15234 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:48.216347 15234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:48.216666 15234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:48.219585 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "5f08f93749934d64813dc9ace6423c79"
format_stamp: "Formatted at 2026-08-10 19:19:48 on dist-test-slave-7lbf"
server_key: "fa3bb8658b56f970b9c93a157f8932b1"
server_key_iv: "bd81d390819c8c1d50061d3fba664fef"
server_key_version: "encryptionkey@0"
I20260810 19:19:48.220355 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "5f08f93749934d64813dc9ace6423c79"
format_stamp: "Formatted at 2026-08-10 19:19:48 on dist-test-slave-7lbf"
server_key: "fa3bb8658b56f970b9c93a157f8932b1"
server_key_iv: "bd81d390819c8c1d50061d3fba664fef"
server_key_version: "encryptionkey@0"
I20260810 19:19:48.225497 15234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260810 19:19:48.228888 15253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:48.230226 15234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:19:48.230382 15234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "5f08f93749934d64813dc9ace6423c79"
format_stamp: "Formatted at 2026-08-10 19:19:48 on dist-test-slave-7lbf"
server_key: "fa3bb8658b56f970b9c93a157f8932b1"
server_key_iv: "bd81d390819c8c1d50061d3fba664fef"
server_key_version: "encryptionkey@0"
I20260810 19:19:48.230620 15234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:48.236189 15234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:48.236937 15234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:48.237179 15234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:48.238111 15234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:48.239442 15234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:48.239537 15234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:48.239642 15234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:48.239714 15234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:48.269765 15234 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34047
I20260810 19:19:48.269812 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34047 every 8 connection(s)
I20260810 19:19:48.271096 15234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:48.275918 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 15234
I20260810 19:19:48.276108 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:48.276712 7340 external_mini_cluster.cc:1428] Setting key d011924fa17cd35a93e3103f55a3189b
I20260810 19:19:48.279301 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:48.273566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:38717 (local address 127.7.43.62:39759)
0810 19:19:48.273993 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:48.273999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:48.275261 (+ 1262us) server_negotiation.cc:407] Connection header received
0810 19:19:48.276989 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:48.276995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:48.277060 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:48.277156 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:48.278567 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:48.278574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:48.278579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:48.278597 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:48.278638 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:48.278649 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:48.278746 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:48.278930 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 19:19:48.279003 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":186,"threads_started":1}
I20260810 19:19:48.279457 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39759
--builtin_ntp_servers=127.7.43.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 19:19:48.280292 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:48.274173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39759 (local address 127.7.43.1:38717)
0810 19:19:48.275109 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:48.275145 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:48.276719 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:48.277395 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:48.277405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:48.277564 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:48.278233 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:48.278236 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:48.278285 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:48.278399 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:48.278733 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:48.278744 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:48.278752 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:19:48.279010 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 19:19:48.279255 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
I20260810 19:19:48.281680 15432 heartbeater.cc:347] Connected to a master server at 127.7.43.62:39759
I20260810 19:19:48.281947 15432 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:48.282503 15432 heartbeater.cc:511] Master 127.7.43.62:39759 requested a full tablet report, sending...
I20260810 19:19:48.284384 15124 ts_manager.cc:194] Registered new tserver with Master: 5f08f93749934d64813dc9ace6423c79 (127.7.43.1:34047)
W20260810 19:19:48.608078 15437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:48.608352 15437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:48.608458 15437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:48.615698 15437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:48.615832 15437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:48.616019 15437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:48.625433 15437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:48.626781 15437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:48.628549 15437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:48.641867 15442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:48.641964 15448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:48.641930 15443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:48.643586 15437 server_base.cc:1034] running on GCE node
I20260810 19:19:48.644227 15437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:48.645192 15437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:48.646401 15437 hybrid_clock.cc:648] HybridClock initialized: now 1786389588646341 us; error 45 us; skew 500 ppm
I20260810 19:19:48.646715 15437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:48.649051 15437 webserver.cc:460] Webserver started at http://127.7.43.2:43421/ using document root <none> and password file <none>
I20260810 19:19:48.649801 15437 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:48.649894 15437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:48.650173 15437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:48.653203 15437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "ef98f045011741ea82d50a748b6b0f10"
format_stamp: "Formatted at 2026-08-10 19:19:48 on dist-test-slave-7lbf"
server_key: "bb5df25286fe0a248e8cb9539187b853"
server_key_iv: "0d625bffad2063c572782c3dec7d88ae"
server_key_version: "encryptionkey@0"
I20260810 19:19:48.654029 15437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "ef98f045011741ea82d50a748b6b0f10"
format_stamp: "Formatted at 2026-08-10 19:19:48 on dist-test-slave-7lbf"
server_key: "bb5df25286fe0a248e8cb9539187b853"
server_key_iv: "0d625bffad2063c572782c3dec7d88ae"
server_key_version: "encryptionkey@0"
I20260810 19:19:48.659322 15437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:48.662979 15456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:48.664354 15437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:48.664525 15437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "ef98f045011741ea82d50a748b6b0f10"
format_stamp: "Formatted at 2026-08-10 19:19:48 on dist-test-slave-7lbf"
server_key: "bb5df25286fe0a248e8cb9539187b853"
server_key_iv: "0d625bffad2063c572782c3dec7d88ae"
server_key_version: "encryptionkey@0"
I20260810 19:19:48.664748 15437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:48.670605 15437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:48.671366 15437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:48.671625 15437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:48.672662 15437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:48.674270 15437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:48.674366 15437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:48.674479 15437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:48.674546 15437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:48.705332 15437 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:39323
I20260810 19:19:48.705413 15648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:39323 every 8 connection(s)
I20260810 19:19:48.706735 15437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:48.713951 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 15437
I20260810 19:19:48.714123 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:48.714694 7340 external_mini_cluster.cc:1428] Setting key 9177d878acd4200ea4a69379bbad9279
I20260810 19:19:48.715420 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:48.709229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:40333 (local address 127.7.43.62:39759)
0810 19:19:48.709393 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:48.709397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:48.710878 (+ 1481us) server_negotiation.cc:407] Connection header received
0810 19:19:48.712560 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:48.712584 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:48.712640 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:48.712724 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:48.714651 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:48.714656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:48.714659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:48.714676 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:48.714711 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:48.714722 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:48.714847 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:48.715050 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 19:19:48.715124 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:19:48.716488 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:48.709764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39759 (local address 127.7.43.2:40333)
0810 19:19:48.710726 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:48.710763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:48.712298 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:48.712973 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:48.712981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:48.713076 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:48.714277 (+ 1201us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:48.714281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:48.714335 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:48.714450 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:48.714840 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:48.714852 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:48.714860 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:19:48.715125 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 19:19:48.715387 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":286,"threads_started":1}
I20260810 19:19:48.717483 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39759
--builtin_ntp_servers=127.7.43.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 19:19:48.718044 15649 heartbeater.cc:347] Connected to a master server at 127.7.43.62:39759
I20260810 19:19:48.718354 15649 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:48.719062 15649 heartbeater.cc:511] Master 127.7.43.62:39759 requested a full tablet report, sending...
I20260810 19:19:48.720598 15124 ts_manager.cc:194] Registered new tserver with Master: ef98f045011741ea82d50a748b6b0f10 (127.7.43.2:39323)
W20260810 19:19:49.051838 15653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:49.052150 15653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:49.052287 15653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:49.059530 15653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:49.059651 15653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:49.059794 15653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:49.069059 15653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:49.070372 15653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:49.072230 15653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:49.084906 15668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:49.084977 15664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:49.085009 15663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:49.085807 15653 server_base.cc:1034] running on GCE node
I20260810 19:19:49.086428 15653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:49.087340 15653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:49.088544 15653 hybrid_clock.cc:648] HybridClock initialized: now 1786389589088482 us; error 50 us; skew 500 ppm
I20260810 19:19:49.088857 15653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:49.091071 15653 webserver.cc:460] Webserver started at http://127.7.43.3:46303/ using document root <none> and password file <none>
I20260810 19:19:49.091758 15653 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:49.091856 15653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:49.092168 15653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:49.095074 15653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "dc626523cb104be8af4ebae6fdb52011"
format_stamp: "Formatted at 2026-08-10 19:19:49 on dist-test-slave-7lbf"
server_key: "20c01a595f888edb3510758bcc487354"
server_key_iv: "98513075e19ca845462efe61e75f9aa2"
server_key_version: "encryptionkey@0"
I20260810 19:19:49.095844 15653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "dc626523cb104be8af4ebae6fdb52011"
format_stamp: "Formatted at 2026-08-10 19:19:49 on dist-test-slave-7lbf"
server_key: "20c01a595f888edb3510758bcc487354"
server_key_iv: "98513075e19ca845462efe61e75f9aa2"
server_key_version: "encryptionkey@0"
I20260810 19:19:49.100776 15653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 19:19:49.104185 15679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:49.105600 15653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:19:49.105759 15653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "dc626523cb104be8af4ebae6fdb52011"
format_stamp: "Formatted at 2026-08-10 19:19:49 on dist-test-slave-7lbf"
server_key: "20c01a595f888edb3510758bcc487354"
server_key_iv: "98513075e19ca845462efe61e75f9aa2"
server_key_version: "encryptionkey@0"
I20260810 19:19:49.106006 15653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:49.111574 15653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:49.112370 15653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:49.112622 15653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:49.113600 15653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:49.115012 15653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:49.115108 15653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:49.115216 15653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:49.115284 15653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:49.145285 15653 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:39245
I20260810 19:19:49.145339 15866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:39245 every 8 connection(s)
I20260810 19:19:49.146636 15653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:49.154513 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 15653
I20260810 19:19:49.154570 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.149151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:35377 (local address 127.7.43.62:39759)
0810 19:19:49.149319 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.149323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.150780 (+ 1457us) server_negotiation.cc:407] Connection header received
0810 19:19:49.152432 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.152438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.152503 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.152590 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.153865 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.153871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.153876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.153892 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.153932 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.153943 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.154060 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.154287 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.154342 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:19:49.154708 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:49.155263 7340 external_mini_cluster.cc:1428] Setting key 0aea307375a2a4f11f3a5fa1e662597e
I20260810 19:19:49.155794 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.149696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39759 (local address 127.7.43.3:35377)
0810 19:19:49.150629 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.150671 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:49.152174 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:49.152830 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:49.152838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:49.152931 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:49.153575 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:49.153577 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:49.153615 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:49.153704 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:49.154031 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:49.154042 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:49.154049 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:19:49.154356 (+ 307us) client_negotiation.cc:240] Negotiation successful
0810 19:19:49.154629 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":271,"threads_started":1}
I20260810 19:19:49.157119 15870 heartbeater.cc:347] Connected to a master server at 127.7.43.62:39759
I20260810 19:19:49.157411 15870 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:49.158097 15870 heartbeater.cc:511] Master 127.7.43.62:39759 requested a full tablet report, sending...
I20260810 19:19:49.159420 15124 ts_manager.cc:194] Registered new tserver with Master: dc626523cb104be8af4ebae6fdb52011 (127.7.43.3:39245)
I20260810 19:19:49.170373 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:19:49.176822 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.174642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.7.43.62:39759)
0810 19:19:49.174801 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.174805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.175299 (+ 494us) server_negotiation.cc:407] Connection header received
0810 19:19:49.175446 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.175450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.175499 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.175579 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.176337 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.176341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.176344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.176358 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.176388 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.176399 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.176505 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.176571 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.176616 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:19:49.180323 15124 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:19:49.187003 15124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:19:49.220227 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.198777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39245 (local address 127.0.0.1:40522)
0810 19:19:49.211514 (+ 12737us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.211564 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:49.211753 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:49.217331 (+ 5578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:49.217341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:49.217461 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:49.218801 (+ 1340us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:49.218805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:49.218836 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:49.218932 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:49.219414 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:49.219439 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:49.219449 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 19:19:49.219642 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 19:19:49.219762 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12403,"thread_start_us":20768,"threads_started":1}
I20260810 19:19:49.221073 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.198942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.7.43.3:39245)
0810 19:19:49.215379 (+ 16437us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.215389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.215442 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:19:49.215528 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.215537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.215802 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.216238 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.219148 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.219160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.219178 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.219199 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.219277 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.219299 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.220469 (+ 1170us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.220554 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.220665 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16233,"thread_start_us":17001,"threads_started":1}
I20260810 19:19:49.222486 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.200407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42734)
0810 19:19:49.219854 (+ 19447us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.219872 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:49.220007 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:49.220783 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:49.220788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:49.220811 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:49.221382 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:49.221385 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:49.221401 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:49.221453 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:49.221822 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:49.221827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:49.221832 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:19:49.221955 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 19:19:49.222006 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19315,"thread_start_us":17741,"threads_started":1}
I20260810 19:19:49.222780 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.200710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.7.43.1:34047)
0810 19:19:49.217788 (+ 17078us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.217797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.220011 (+ 2214us) server_negotiation.cc:407] Connection header received
0810 19:19:49.220216 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.220225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.220456 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.220571 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.221588 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.221596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.221610 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.221637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.221692 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.221712 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.222299 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.222385 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.222482 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16875,"thread_start_us":17085,"threads_started":1}
I20260810 19:19:49.223034 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.198955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:39323 (local address 127.0.0.1:34210)
0810 19:19:49.218438 (+ 19483us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.218467 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:49.218573 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:49.219650 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:49.219654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:49.219683 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:49.220663 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:49.220667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:49.220684 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:49.220727 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:49.222005 (+ 1278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:49.222010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:49.222015 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:19:49.222152 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 19:19:49.222201 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19337,"thread_start_us":12051,"threads_started":1}
I20260810 19:19:49.223523 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.199364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.7.43.2:39323)
0810 19:19:49.219072 (+ 19708us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.219079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.219113 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:19:49.219182 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.219189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.219427 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.220207 (+ 780us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.221772 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.221780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.221793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.221809 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.221869 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.221888 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.223081 (+ 1193us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.223157 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.223248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19538,"thread_start_us":19415,"threads_started":1}
I20260810 19:19:49.225266 15756 tablet_service.cc:1467] Processing CreateTablet for tablet 9af5f56202bc4872a63968dd3f2488e4 (DEFAULT_TABLE table=test-table [id=ce3d26ebb2ce4a19b43fcbfec3f2d4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:49.226537 15756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af5f56202bc4872a63968dd3f2488e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:49.226997 15542 tablet_service.cc:1467] Processing CreateTablet for tablet 9af5f56202bc4872a63968dd3f2488e4 (DEFAULT_TABLE table=test-table [id=ce3d26ebb2ce4a19b43fcbfec3f2d4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:49.227005 15329 tablet_service.cc:1467] Processing CreateTablet for tablet 9af5f56202bc4872a63968dd3f2488e4 (DEFAULT_TABLE table=test-table [id=ce3d26ebb2ce4a19b43fcbfec3f2d4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:19:49.228402 15542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af5f56202bc4872a63968dd3f2488e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:49.228401 15329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af5f56202bc4872a63968dd3f2488e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:49.240012 15897 tablet_bootstrap.cc:492] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Bootstrap starting.
I20260810 19:19:49.243352 15897 tablet_bootstrap.cc:654] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:49.244469 15901 tablet_bootstrap.cc:492] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Bootstrap starting.
I20260810 19:19:49.244917 15897 log.cc:824] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:49.247648 15899 tablet_bootstrap.cc:492] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Bootstrap starting.
I20260810 19:19:49.248194 15901 tablet_bootstrap.cc:654] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:49.249234 15897 tablet_bootstrap.cc:492] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: No bootstrap required, opened a new log
I20260810 19:19:49.249490 15897 ts_tablet_manager.cc:1397] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 19:19:49.249473 15901 log.cc:824] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:49.251681 15899 tablet_bootstrap.cc:654] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:49.252822 15899 log.cc:824] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:49.253983 15901 tablet_bootstrap.cc:492] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: No bootstrap required, opened a new log
I20260810 19:19:49.254240 15901 ts_tablet_manager.cc:1397] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260810 19:19:49.256053 15899 tablet_bootstrap.cc:492] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: No bootstrap required, opened a new log
I20260810 19:19:49.256305 15899 ts_tablet_manager.cc:1397] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 19:19:49.258635 15897 raft_consensus.cc:348] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.259078 15897 raft_consensus.cc:374] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:49.259164 15897 raft_consensus.cc:729] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc626523cb104be8af4ebae6fdb52011, State: Initialized, Role: FOLLOWER
I20260810 19:19:49.259780 15897 consensus_queue.cc:260] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.260959 15870 heartbeater.cc:503] Master 127.7.43.62:39759 was elected leader, sending a full tablet report...
I20260810 19:19:49.261641 15897 ts_tablet_manager.cc:1428] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 19:19:49.262967 15901 raft_consensus.cc:348] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.263353 15901 raft_consensus.cc:374] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:49.263453 15901 raft_consensus.cc:729] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef98f045011741ea82d50a748b6b0f10, State: Initialized, Role: FOLLOWER
I20260810 19:19:49.264036 15901 consensus_queue.cc:260] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.265132 15649 heartbeater.cc:503] Master 127.7.43.62:39759 was elected leader, sending a full tablet report...
I20260810 19:19:49.265774 15901 ts_tablet_manager.cc:1428] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260810 19:19:49.265938 15899 raft_consensus.cc:348] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.266286 15899 raft_consensus.cc:374] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:49.266376 15899 raft_consensus.cc:729] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f08f93749934d64813dc9ace6423c79, State: Initialized, Role: FOLLOWER
I20260810 19:19:49.266964 15899 consensus_queue.cc:260] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.268644 15432 heartbeater.cc:503] Master 127.7.43.62:39759 was elected leader, sending a full tablet report...
I20260810 19:19:49.270068 15899 ts_tablet_manager.cc:1428] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260810 19:19:49.273245 15434 tablet.cc:2369] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:49.337078 15909 raft_consensus.cc:482] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:19:49.337390 15909 raft_consensus.cc:504] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.339244 15909 leader_election.cc:288] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc626523cb104be8af4ebae6fdb52011 (127.7.43.3:39245), 5f08f93749934d64813dc9ace6423c79 (127.7.43.1:34047)
I20260810 19:19:49.342587 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.339810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:53993 (local address 127.7.43.3:39245)
0810 19:19:49.339983 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.339989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.340730 (+ 741us) server_negotiation.cc:407] Connection header received
0810 19:19:49.340893 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.340897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.340955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.341035 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.342016 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.342021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.342026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.342040 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.342072 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.342083 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.342191 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.342270 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.342334 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:19:49.342676 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.339627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:34403 (local address 127.7.43.1:34047)
0810 19:19:49.339812 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.339818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.340994 (+ 1176us) server_negotiation.cc:407] Connection header received
0810 19:19:49.341124 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.341128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.341177 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.341256 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.342103 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.342109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.342112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.342127 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.342155 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.342167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.342261 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.342352 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.342410 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:19:49.342854 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.339654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39245 (local address 127.7.43.2:53993)
0810 19:19:49.340610 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.340633 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:49.340770 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:49.341083 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:49.341087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:49.341111 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:49.341789 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:49.341793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:49.341811 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:49.341861 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:49.342168 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:49.342173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:49.342178 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:19:49.342322 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 19:19:49.342370 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":535,"threads_started":1}
I20260810 19:19:49.342972 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.340108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.7.43.2:34403)
0810 19:19:49.340879 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.340893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:49.341001 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:49.341308 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:49.341312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:49.341339 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:49.341918 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:19:49.341920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:49.341933 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:19:49.341970 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:49.342249 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:49.342253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:49.342257 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:19:49.342397 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 19:19:49.342463 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":461,"threads_started":1}
I20260810 19:19:49.343791 15789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af5f56202bc4872a63968dd3f2488e4" candidate_uuid: "ef98f045011741ea82d50a748b6b0f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc626523cb104be8af4ebae6fdb52011" is_pre_election: true
I20260810 19:19:49.343854 15359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af5f56202bc4872a63968dd3f2488e4" candidate_uuid: "ef98f045011741ea82d50a748b6b0f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f08f93749934d64813dc9ace6423c79" is_pre_election: true
I20260810 19:19:49.344271 15789 raft_consensus.cc:2454] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef98f045011741ea82d50a748b6b0f10 in term 0.
I20260810 19:19:49.344316 15359 raft_consensus.cc:2454] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef98f045011741ea82d50a748b6b0f10 in term 0.
I20260810 19:19:49.344888 15461 leader_election.cc:302] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc626523cb104be8af4ebae6fdb52011, ef98f045011741ea82d50a748b6b0f10; no voters:
I20260810 19:19:49.345233 15909 raft_consensus.cc:2781] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:19:49.345357 15909 raft_consensus.cc:482] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:19:49.345463 15909 raft_consensus.cc:3037] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:49.347630 15909 raft_consensus.cc:504] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.348187 15909 leader_election.cc:288] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [CANDIDATE]: Term 1 election: Requested vote from peers dc626523cb104be8af4ebae6fdb52011 (127.7.43.3:39245), 5f08f93749934d64813dc9ace6423c79 (127.7.43.1:34047)
I20260810 19:19:49.348591 15789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af5f56202bc4872a63968dd3f2488e4" candidate_uuid: "ef98f045011741ea82d50a748b6b0f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc626523cb104be8af4ebae6fdb52011"
I20260810 19:19:49.348626 15359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af5f56202bc4872a63968dd3f2488e4" candidate_uuid: "ef98f045011741ea82d50a748b6b0f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f08f93749934d64813dc9ace6423c79"
I20260810 19:19:49.348759 15789 raft_consensus.cc:3037] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:49.348796 15359 raft_consensus.cc:3037] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:49.351006 15789 raft_consensus.cc:2454] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef98f045011741ea82d50a748b6b0f10 in term 1.
I20260810 19:19:49.351120 15359 raft_consensus.cc:2454] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef98f045011741ea82d50a748b6b0f10 in term 1.
I20260810 19:19:49.351377 15461 leader_election.cc:302] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc626523cb104be8af4ebae6fdb52011, ef98f045011741ea82d50a748b6b0f10; no voters:
I20260810 19:19:49.351644 15909 raft_consensus.cc:2781] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:49.352044 15909 raft_consensus.cc:686] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 1 LEADER]: Becoming Leader. State: Replica: ef98f045011741ea82d50a748b6b0f10, State: Running, Role: LEADER
I20260810 19:19:49.352494 15909 consensus_queue.cc:237] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } }
I20260810 19:19:49.356808 15123 catalog_manager.cc:5495] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef98f045011741ea82d50a748b6b0f10 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "ef98f045011741ea82d50a748b6b0f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef98f045011741ea82d50a748b6b0f10" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 } health_report { overall_health: UNKNOWN } } }
I20260810 19:19:49.378543 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.376653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.7.43.2:39323)
0810 19:19:49.376852 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.376856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.376905 (+ 49us) server_negotiation.cc:407] Connection header received
0810 19:19:49.377091 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.377095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.377175 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.377252 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.378013 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.378017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.378020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.378035 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.378065 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.378076 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.378161 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.378216 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.378258 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:19:49.386584 15789 raft_consensus.cc:1261] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 1 FOLLOWER]: Refusing update from remote peer ef98f045011741ea82d50a748b6b0f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:49.386584 15359 raft_consensus.cc:1261] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 1 FOLLOWER]: Refusing update from remote peer ef98f045011741ea82d50a748b6b0f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:19:49.387588 15909 consensus_queue.cc:1035] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f08f93749934d64813dc9ace6423c79" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:19:49.387880 15916 consensus_queue.cc:1035] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc626523cb104be8af4ebae6fdb52011" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:19:49.398160 15871 tablet.cc:2369] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:19:49.400098 15921 mvcc.cc:205] Tried to move back new op lower bound from 7317051758117994496 to 7317051757991497728. Current Snapshot: MvccSnapshot[applied={T|T < 7317051758117994496}]
I20260810 19:19:49.415365 15123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:49.415652 15123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:19:49.419021 15123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68bde36af8324c048c0b2f9e798069e9: Sending DeleteTablet for 3 replicas of tablet 9af5f56202bc4872a63968dd3f2488e4
I20260810 19:19:49.420188 15542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af5f56202bc4872a63968dd3f2488e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:49 UTC) from {username='slave'} at 127.0.0.1:34210
I20260810 19:19:49.420367 15756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af5f56202bc4872a63968dd3f2488e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:49 UTC) from {username='slave'} at 127.0.0.1:40522
I20260810 19:19:49.423946 15329 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af5f56202bc4872a63968dd3f2488e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:19:49 UTC) from {username='slave'} at 127.0.0.1:42734
I20260810 19:19:49.426558 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 15234
I20260810 19:19:49.432776 15940 tablet_replica.cc:330] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: stopping tablet replica
I20260810 19:19:49.433451 15940 raft_consensus.cc:2229] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:49.434095 15940 raft_consensus.cc:2258] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:49.435022 15945 tablet_replica.cc:330] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: stopping tablet replica
I20260810 19:19:49.435451 15945 raft_consensus.cc:2229] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:19:49.435953 15945 raft_consensus.cc:2258] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:49.436688 15940 ts_tablet_manager.cc:1905] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:49.438122 15945 ts_tablet_manager.cc:1905] T 9af5f56202bc4872a63968dd3f2488e4 P 5f08f93749934d64813dc9ace6423c79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:49.439270 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.428242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.7.43.1:34047)
0810 19:19:49.428451 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.428458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.436694 (+ 8236us) server_negotiation.cc:407] Connection header received
0810 19:19:49.437015 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.437021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.437112 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.437188 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.438429 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.438435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.438441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.438461 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.438498 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.438515 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.438620 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.438872 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.438952 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:19:49.439538 15939 tablet_replica.cc:330] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: stopping tablet replica
I20260810 19:19:49.439991 15939 raft_consensus.cc:2229] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:19:49.440807 15939 raft_consensus.cc:2258] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:19:49.442265 15426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:49.442957 15939 ts_tablet_manager.cc:1905] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:19:49.445072 15940 ts_tablet_manager.cc:1918] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:49.445245 15940 log.cc:1196] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/9af5f56202bc4872a63968dd3f2488e4
I20260810 19:19:49.445809 15940 ts_tablet_manager.cc:1939] T 9af5f56202bc4872a63968dd3f2488e4 P dc626523cb104be8af4ebae6fdb52011: Deleting consensus metadata
I20260810 19:19:49.447700 15101 catalog_manager.cc:4841] TS dc626523cb104be8af4ebae6fdb52011 (127.7.43.3:39245): tablet 9af5f56202bc4872a63968dd3f2488e4 (table test-table [id=ce3d26ebb2ce4a19b43fcbfec3f2d4ea]) successfully deleted
I20260810 19:19:49.451730 15939 ts_tablet_manager.cc:1918] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:19:49.452067 15939 log.cc:1196] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/9af5f56202bc4872a63968dd3f2488e4
I20260810 19:19:49.452757 15939 ts_tablet_manager.cc:1939] T 9af5f56202bc4872a63968dd3f2488e4 P ef98f045011741ea82d50a748b6b0f10: Deleting consensus metadata
I20260810 19:19:49.454247 15102 catalog_manager.cc:4841] TS ef98f045011741ea82d50a748b6b0f10 (127.7.43.2:39323): tablet 9af5f56202bc4872a63968dd3f2488e4 (table test-table [id=ce3d26ebb2ce4a19b43fcbfec3f2d4ea]) successfully deleted
I20260810 19:19:49.567080 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 15234
I20260810 19:19:49.601372 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 15437
W20260810 19:19:49.601372 15099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:34047 (error 108)
I20260810 19:19:49.604096 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.603489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42754)
0810 19:19:49.603753 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.603927 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260810 19:19:49.604370 15099 catalog_manager.cc:4584] TS 5f08f93749934d64813dc9ace6423c79 (127.7.43.1:34047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9af5f56202bc4872a63968dd3f2488e4: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
I20260810 19:19:49.605108 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.602253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.7.43.2:39323)
0810 19:19:49.602475 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.602483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.603083 (+ 600us) server_negotiation.cc:407] Connection header received
0810 19:19:49.603172 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.603175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.603226 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.603305 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.604412 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.604418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.604422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.604441 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.604477 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.604492 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.604609 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.604800 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.604860 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:19:49.605860 15640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:49.658154 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.657577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42756)
0810 19:19:49.657811 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.657942 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:19:49.659189 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.658808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42768)
0810 19:19:49.658958 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.659050 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:19:49.718011 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 15437
I20260810 19:19:49.727262 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.726761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42772)
0810 19:19:49.726972 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.727101 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 19:19:49.728322 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.727959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42786)
0810 19:19:49.728085 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.728161 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 19:19:49.743647 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 15653
I20260810 19:19:49.746132 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.744376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.7.43.3:39245)
0810 19:19:49.744537 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:49.744541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:49.744589 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:19:49.744723 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:49.744727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:49.744800 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:49.744879 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:49.745677 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:49.745680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:49.745683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:49.745695 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:49.745723 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:49.745733 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:49.745826 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:49.745862 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 19:19:49.745905 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:19:49.746732 15860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:49.809131 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.808622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42798)
0810 19:19:49.808831 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.808932 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 19:19:49.810281 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:49.809869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34047 (local address 127.0.0.1:42810)
0810 19:19:49.810081 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:49.810152 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 19:19:49.833391 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 15653
I20260810 19:19:49.858564 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 15079
I20260810 19:19:49.859709 15190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:49.965219 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 15079
2026-08-10T19:19:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2729 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 19:19:50.012633 7340 mini_oidc.cc:152] Starting JWKS server
I20260810 19:19:50.015690 7340 webserver.cc:460] Webserver started at https://127.0.0.1:43839/ using document root <none> and password file <none>
I20260810 19:19:50.015997 7340 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 19:19:50.016649 7340 webserver.cc:460] Webserver started at http://127.0.0.1:37591/ using document root <none> and password file <none>
2026-08-10T19:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:50Z Disabled control of system clock
I20260810 19:19:50.052212 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:45755
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:45755
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/testchainca.cert with env {}
W20260810 19:19:50.379441 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:50.379810 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:50.379969 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:50.387416 15965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:50.387563 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:50.387638 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:50.387728 15965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:50.387815 15965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:19:50.387915 15965 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:19:50.387979 15965 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43839/jwks.json
I20260810 19:19:50.398283 15965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:45755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:50.399658 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:50.401051 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:50.413842 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:50.413924 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:50.413892 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:50.414705 15965 server_base.cc:1034] running on GCE node
I20260810 19:19:50.415522 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:50.417013 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:50.418231 15965 hybrid_clock.cc:648] HybridClock initialized: now 1786389590418174 us; error 45 us; skew 500 ppm
I20260810 19:19:50.418597 15965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:50.420826 15965 webserver.cc:460] Webserver started at http://127.7.43.62:33819/ using document root <none> and password file <none>
I20260810 19:19:50.421509 15965 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:50.421651 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:50.421986 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:50.425386 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "64a76ddcffec4471a09a325fdd83c629"
format_stamp: "Formatted at 2026-08-10 19:19:50 on dist-test-slave-7lbf"
server_key: "49d72a5216736a57c03682cc39142ea3"
server_key_iv: "b52873b0ec16d08d5dd0b4301e208856"
server_key_version: "encryptionkey@0"
I20260810 19:19:50.426414 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "64a76ddcffec4471a09a325fdd83c629"
format_stamp: "Formatted at 2026-08-10 19:19:50 on dist-test-slave-7lbf"
server_key: "49d72a5216736a57c03682cc39142ea3"
server_key_iv: "b52873b0ec16d08d5dd0b4301e208856"
server_key_version: "encryptionkey@0"
I20260810 19:19:50.431418 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:50.434790 15989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:50.436044 15965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:50.436194 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "64a76ddcffec4471a09a325fdd83c629"
format_stamp: "Formatted at 2026-08-10 19:19:50 on dist-test-slave-7lbf"
server_key: "49d72a5216736a57c03682cc39142ea3"
server_key_iv: "b52873b0ec16d08d5dd0b4301e208856"
server_key_version: "encryptionkey@0"
I20260810 19:19:50.436384 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:50.463160 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:50.463943 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:50.464210 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:50.481323 15965 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:45755
I20260810 19:19:50.481344 16058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:45755 every 8 connection(s)
I20260810 19:19:50.482666 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:50.486332 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:50.492205 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 15965
I20260810 19:19:50.492372 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:50.492998 7340 external_mini_cluster.cc:1428] Setting key 63fd00783c59407dea1ca8e6133e0489
I20260810 19:19:50.497390 16059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629: Bootstrap starting.
I20260810 19:19:50.501580 16059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:50.502758 16059 log.cc:824] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:50.505824 16059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629: No bootstrap required, opened a new log
I20260810 19:19:50.508487 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:50.494942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.7.43.62:45755)
0810 19:19:50.496041 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:50.496057 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:50.496107 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:19:50.498468 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:50.498497 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:50.499115 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:50.499616 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:50.500173 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:50.502128 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:50.503637 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:50.504495 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:50.504671 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:50.505716 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:50.505727 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:50.505745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:50.506471 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:50.506570 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:50.506599 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:50.506718 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:50.506893 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 19:19:50.507215 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":339,"threads_started":1}
I20260810 19:19:50.512168 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:19:50.514338 16059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a76ddcffec4471a09a325fdd83c629" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45755 } }
I20260810 19:19:50.514701 16059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:50.514807 16059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a76ddcffec4471a09a325fdd83c629, State: Initialized, Role: FOLLOWER
I20260810 19:19:50.515389 16059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a76ddcffec4471a09a325fdd83c629" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45755 } }
I20260810 19:19:50.515614 16059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:50.515741 16059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:50.515892 16059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:50.518127 16059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a76ddcffec4471a09a325fdd83c629" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45755 } }
I20260810 19:19:50.518607 16059 leader_election.cc:302] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a76ddcffec4471a09a325fdd83c629; no voters:
I20260810 19:19:50.519171 16059 leader_election.cc:288] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:50.519289 16065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:50.519968 16065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [term 1 LEADER]: Becoming Leader. State: Replica: 64a76ddcffec4471a09a325fdd83c629, State: Running, Role: LEADER
I20260810 19:19:50.520490 16065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a76ddcffec4471a09a325fdd83c629" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45755 } }
I20260810 19:19:50.521231 16059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:50.523537 16066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a76ddcffec4471a09a325fdd83c629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a76ddcffec4471a09a325fdd83c629" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45755 } } }
I20260810 19:19:50.523993 16066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:50.524331 16068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a76ddcffec4471a09a325fdd83c629. Latest consensus state: current_term: 1 leader_uuid: "64a76ddcffec4471a09a325fdd83c629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a76ddcffec4471a09a325fdd83c629" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45755 } } }
I20260810 19:19:50.524631 16068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:50.525907 16079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:50.530337 16079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:50.538604 16079 catalog_manager.cc:1347] Generated new cluster ID: 3785058ddba242989a2784ec3e62c7d8
I20260810 19:19:50.538723 16079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:50.551918 16079 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:50.553179 16079 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:50.568071 16079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a76ddcffec4471a09a325fdd83c629: Generated new TSK 0
I20260810 19:19:50.569126 16079 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 19:19:50.612712 16108 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 19:19:50.612931 16108 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 19:19:50.614259 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:50.592475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.7.43.62:45755)
0810 19:19:50.592674 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:50.592681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:50.593068 (+ 387us) server_negotiation.cc:407] Connection header received
0810 19:19:50.593396 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:50.593402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:50.593502 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:50.593680 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:19:50.594159 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:50.595274 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:50.596598 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:50.597286 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:50.597410 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:50.597909 (+ 499us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:19:50.613554 (+ 15645us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 19:19:50.613963 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 19:19:50.614394 16063 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 19:19:50.615904 16107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:50.592306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45755 (local address 127.0.0.1:36612)
0810 19:19:50.592971 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:50.592987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:50.593133 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:50.593698 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:50.593702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:50.593711 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:19:50.594021 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:50.594029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:50.595390 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:50.595393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:50.596462 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:50.596473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:50.597386 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:50.597423 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:50.597585 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:50.597651 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:19:50.613876 (+ 16225us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 19:19:50.614182 (+ 306us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.43.62:45755: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":335,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.43.62:45755: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 19:19:50.618016 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 15965
I20260810 19:19:50.619082 16053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:50.712015 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 15965
2026-08-10T19:19:50Z chronyd exiting
I20260810 19:19:50.768287 7340 test_util.cc:175] -----------------------------------------------
I20260810 19:19:50.768416 7340 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786389541863099-7340-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (760 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 19:19:50.772440 7340 mini_oidc.cc:152] Starting JWKS server
I20260810 19:19:50.774313 7340 webserver.cc:460] Webserver started at https://127.0.0.1:42059/ using document root <none> and password file <none>
I20260810 19:19:50.774561 7340 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 19:19:50.775219 7340 webserver.cc:460] Webserver started at http://127.0.0.1:38265/ using document root <none> and password file <none>
2026-08-10T19:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:50Z Disabled control of system clock
I20260810 19:19:50.812748 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39269
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39269
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/testchainca.cert with env {}
W20260810 19:19:51.139379 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:51.139678 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:51.139806 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:51.147089 16122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:51.147198 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:51.147255 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:51.147336 16122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:51.147403 16122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:19:51.147490 16122 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:19:51.147547 16122 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42059/jwks.json
I20260810 19:19:51.157737 16122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:51.159053 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:51.160320 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:51.172065 16129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:51.172037 16130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:51.172130 16132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:51.174990 16122 server_base.cc:1034] running on GCE node
I20260810 19:19:51.175765 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:51.177217 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:51.178409 16122 hybrid_clock.cc:648] HybridClock initialized: now 1786389591178358 us; error 41 us; skew 500 ppm
I20260810 19:19:51.178755 16122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:51.186538 16122 webserver.cc:460] Webserver started at http://127.7.43.62:35471/ using document root <none> and password file <none>
I20260810 19:19:51.187263 16122 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:51.187364 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:51.187731 16122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:51.190821 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "4e71591e3da6434faf25773475b0198c"
format_stamp: "Formatted at 2026-08-10 19:19:51 on dist-test-slave-7lbf"
server_key: "9ae4a715bf38b0e58466addcc7d65dbd"
server_key_iv: "2bf440dad3cdf24d5c2f5a997e24170c"
server_key_version: "encryptionkey@0"
I20260810 19:19:51.191614 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "4e71591e3da6434faf25773475b0198c"
format_stamp: "Formatted at 2026-08-10 19:19:51 on dist-test-slave-7lbf"
server_key: "9ae4a715bf38b0e58466addcc7d65dbd"
server_key_iv: "2bf440dad3cdf24d5c2f5a997e24170c"
server_key_version: "encryptionkey@0"
I20260810 19:19:51.196764 16122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 19:19:51.200201 16144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:51.201670 16122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:19:51.201833 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "4e71591e3da6434faf25773475b0198c"
format_stamp: "Formatted at 2026-08-10 19:19:51 on dist-test-slave-7lbf"
server_key: "9ae4a715bf38b0e58466addcc7d65dbd"
server_key_iv: "2bf440dad3cdf24d5c2f5a997e24170c"
server_key_version: "encryptionkey@0"
I20260810 19:19:51.202041 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:51.215492 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:51.216310 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:51.216578 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:51.234421 16122 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:39269
I20260810 19:19:51.234432 16225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:39269 every 8 connection(s)
I20260810 19:19:51.235761 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:51.239578 16226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:51.242203 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 16122
I20260810 19:19:51.242435 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:51.243234 7340 external_mini_cluster.cc:1428] Setting key b0ce8d3f95129acfae4c87f6edfc7797
I20260810 19:19:51.251510 16226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c: Bootstrap starting.
I20260810 19:19:51.254575 16226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:51.255774 16226 log.cc:824] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:51.256254 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:51.245328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.7.43.62:39269)
0810 19:19:51.246387 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:51.246398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:51.246446 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:19:51.248065 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:51.248088 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:51.248552 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:51.248904 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:51.249492 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.250778 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.252080 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.252744 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.252942 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:51.253999 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:51.254007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:51.254020 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:51.254535 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:51.254632 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:51.254652 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:51.254778 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:51.254988 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 19:19:51.255290 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":397,"threads_started":1}
I20260810 19:19:51.258752 16226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c: No bootstrap required, opened a new log
I20260810 19:19:51.259917 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:19:51.266973 16226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e71591e3da6434faf25773475b0198c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39269 } }
I20260810 19:19:51.267330 16226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:51.267422 16226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e71591e3da6434faf25773475b0198c, State: Initialized, Role: FOLLOWER
I20260810 19:19:51.267951 16226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e71591e3da6434faf25773475b0198c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39269 } }
I20260810 19:19:51.268169 16226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:51.268301 16226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:51.268464 16226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:51.270695 16226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e71591e3da6434faf25773475b0198c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39269 } }
I20260810 19:19:51.271190 16226 leader_election.cc:302] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e71591e3da6434faf25773475b0198c; no voters:
I20260810 19:19:51.271751 16226 leader_election.cc:288] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:51.271875 16235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:51.272545 16235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [term 1 LEADER]: Becoming Leader. State: Replica: 4e71591e3da6434faf25773475b0198c, State: Running, Role: LEADER
I20260810 19:19:51.273061 16235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e71591e3da6434faf25773475b0198c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39269 } }
I20260810 19:19:51.273759 16226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:51.275616 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e71591e3da6434faf25773475b0198c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e71591e3da6434faf25773475b0198c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39269 } } }
I20260810 19:19:51.275950 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:51.276810 16237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e71591e3da6434faf25773475b0198c. Latest consensus state: current_term: 1 leader_uuid: "4e71591e3da6434faf25773475b0198c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e71591e3da6434faf25773475b0198c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39269 } } }
I20260810 19:19:51.277112 16237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:51.277809 16240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:51.282876 16240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:51.291697 16240 catalog_manager.cc:1347] Generated new cluster ID: 54366b20fd3e47aa98ab28338c533504
I20260810 19:19:51.291802 16240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:51.318045 16240 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:51.319340 16240 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:51.326953 16240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e71591e3da6434faf25773475b0198c: Generated new TSK 0
I20260810 19:19:51.327950 16240 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 19:19:51.367492 16277 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 19:19:51.367602 16277 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 19:19:51.368835 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:51.349024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.7.43.62:39269)
0810 19:19:51.349211 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:51.349217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:51.349602 (+ 385us) server_negotiation.cc:407] Connection header received
0810 19:19:51.349823 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:51.349829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:51.349917 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:51.350084 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:19:51.350462 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.351595 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.352932 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.353625 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.353743 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:51.354210 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:19:51.368157 (+ 13947us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 19:19:51.368527 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 19:19:51.368875 16274 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:51.348876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39269 (local address 127.0.0.1:33140)
0810 19:19:51.349490 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:51.349505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:51.349658 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:51.350079 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:51.350082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:51.350091 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:19:51.350315 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:51.350323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.351703 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.351706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:51.352794 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:51.352805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.353724 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.353780 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:51.353922 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:51.353953 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:19:51.368382 (+ 14429us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 19:19:51.368592 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.43.62:39269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":332,"threads_started":1}
W20260810 19:19:51.368971 16233 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260810 19:19:51.369951 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 16122
I20260810 19:19:51.371100 16218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:51.464183 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 16122
2026-08-10T19:19:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (756 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 19:19:51.528594 7340 mini_oidc.cc:152] Starting JWKS server
I20260810 19:19:51.530555 7340 webserver.cc:460] Webserver started at https://127.0.0.1:44289/ using document root <none> and password file <none>
I20260810 19:19:51.530771 7340 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 19:19:51.531389 7340 webserver.cc:460] Webserver started at http://127.0.0.1:36749/ using document root <none> and password file <none>
2026-08-10T19:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:51Z Disabled control of system clock
I20260810 19:19:51.566798 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44149
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:44149
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44289/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 19:19:51.890477 16286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:51.890834 16286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:51.890945 16286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:51.898396 16286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:51.898511 16286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:51.898557 16286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:51.898614 16286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:51.898658 16286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:19:51.898725 16286 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:19:51.898764 16286 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44289/jwks.json
I20260810 19:19:51.908047 16286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:44149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44289/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:51.909335 16286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:51.911237 16286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:51.923112 16292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:51.923142 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:51.923182 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:51.923982 16286 server_base.cc:1034] running on GCE node
I20260810 19:19:51.924782 16286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:51.926302 16286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:51.927520 16286 hybrid_clock.cc:648] HybridClock initialized: now 1786389591927469 us; error 44 us; skew 500 ppm
I20260810 19:19:51.927862 16286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:51.930085 16286 webserver.cc:460] Webserver started at http://127.7.43.62:37381/ using document root <none> and password file <none>
I20260810 19:19:51.930783 16286 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:51.930888 16286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:51.931224 16286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:51.934180 16286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "c177ee1e18ba41378948a85b3e79e428"
format_stamp: "Formatted at 2026-08-10 19:19:51 on dist-test-slave-7lbf"
server_key: "9212eef177fbc648431c9e2502b89288"
server_key_iv: "2a6453510b930594e3cf137244353b5f"
server_key_version: "encryptionkey@0"
I20260810 19:19:51.934974 16286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "c177ee1e18ba41378948a85b3e79e428"
format_stamp: "Formatted at 2026-08-10 19:19:51 on dist-test-slave-7lbf"
server_key: "9212eef177fbc648431c9e2502b89288"
server_key_iv: "2a6453510b930594e3cf137244353b5f"
server_key_version: "encryptionkey@0"
I20260810 19:19:51.940166 16286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:19:51.943677 16302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:51.945006 16286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:19:51.945174 16286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "c177ee1e18ba41378948a85b3e79e428"
format_stamp: "Formatted at 2026-08-10 19:19:51 on dist-test-slave-7lbf"
server_key: "9212eef177fbc648431c9e2502b89288"
server_key_iv: "2a6453510b930594e3cf137244353b5f"
server_key_version: "encryptionkey@0"
I20260810 19:19:51.945374 16286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:51.964299 16286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:51.965198 16286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:51.965476 16286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:51.983351 16286 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:44149
I20260810 19:19:51.983374 16399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:44149 every 8 connection(s)
I20260810 19:19:51.984750 16286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:51.986423 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 16286
I20260810 19:19:51.986637 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:51.987501 7340 external_mini_cluster.cc:1428] Setting key b838c4db5dd1ec626936b40f2892b8a2
I20260810 19:19:51.989565 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:52.001092 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:51.989432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.7.43.62:44149)
0810 19:19:51.990497 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:51.990508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:51.990564 (+ 56us) server_negotiation.cc:407] Connection header received
0810 19:19:51.992151 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:51.992173 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:51.992644 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:51.992974 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:51.993598 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.994963 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.996774 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:51.997415 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:51.997600 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:51.998758 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:51.998766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:51.998780 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:51.999333 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:51.999407 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:51.999431 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:51.999528 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:51.999748 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 19:19:52.000109 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":384,"threads_started":1}
I20260810 19:19:52.002498 16400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428: Bootstrap starting.
I20260810 19:19:52.004931 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:19:52.005796 16400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:52.006768 16400 log.cc:824] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:52.009683 16400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428: No bootstrap required, opened a new log
I20260810 19:19:52.017022 16400 raft_consensus.cc:348] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c177ee1e18ba41378948a85b3e79e428" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44149 } }
I20260810 19:19:52.017380 16400 raft_consensus.cc:374] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:52.017477 16400 raft_consensus.cc:729] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c177ee1e18ba41378948a85b3e79e428, State: Initialized, Role: FOLLOWER
I20260810 19:19:52.018036 16400 consensus_queue.cc:260] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c177ee1e18ba41378948a85b3e79e428" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44149 } }
I20260810 19:19:52.018257 16400 raft_consensus.cc:388] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:52.018380 16400 raft_consensus.cc:482] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:52.018546 16400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:52.020826 16400 raft_consensus.cc:504] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c177ee1e18ba41378948a85b3e79e428" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44149 } }
I20260810 19:19:52.021339 16400 leader_election.cc:302] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c177ee1e18ba41378948a85b3e79e428; no voters:
I20260810 19:19:52.022006 16400 leader_election.cc:288] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:52.022095 16406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:52.022790 16406 raft_consensus.cc:686] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [term 1 LEADER]: Becoming Leader. State: Replica: c177ee1e18ba41378948a85b3e79e428, State: Running, Role: LEADER
I20260810 19:19:52.023322 16406 consensus_queue.cc:237] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c177ee1e18ba41378948a85b3e79e428" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44149 } }
I20260810 19:19:52.024086 16400 sys_catalog.cc:564] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:52.025815 16408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c177ee1e18ba41378948a85b3e79e428. Latest consensus state: current_term: 1 leader_uuid: "c177ee1e18ba41378948a85b3e79e428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c177ee1e18ba41378948a85b3e79e428" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44149 } } }
I20260810 19:19:52.026266 16408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:52.027156 16414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:52.027578 16407 sys_catalog.cc:455] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c177ee1e18ba41378948a85b3e79e428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c177ee1e18ba41378948a85b3e79e428" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44149 } } }
I20260810 19:19:52.027887 16407 sys_catalog.cc:458] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:52.033740 16414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:52.046689 16414 catalog_manager.cc:1347] Generated new cluster ID: 4b5b8e9d63ad4ff392646be32fd79ffa
I20260810 19:19:52.046800 16414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:52.070737 16414 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:52.072681 16414 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:52.094632 16414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c177ee1e18ba41378948a85b3e79e428: Generated new TSK 0
I20260810 19:19:52.095752 16414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:19:52.126191 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:52.117838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.7.43.62:44149)
0810 19:19:52.118040 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:52.118050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:52.118391 (+ 341us) server_negotiation.cc:407] Connection header received
0810 19:19:52.118623 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:52.118633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:52.118727 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:52.118900 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:19:52.119290 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.120401 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.121769 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.122497 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.122620 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:52.123090 (+ 470us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:19:52.125545 (+ 2455us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44289/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 19:19:52.125869 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44289/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 19:19:52.126322 16404 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44289/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 19:19:52.126736 16440 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:52.117700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44149 (local address 127.0.0.1:35030)
0810 19:19:52.118293 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:52.118309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:52.118431 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:52.118896 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:52.118899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:52.118908 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:19:52.119149 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:52.119157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.120518 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.120522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:52.121633 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:52.121645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.122600 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.122665 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:52.122787 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:52.122820 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:19:52.125757 (+ 2937us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44289/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 19:19:52.126038 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.43.62:44149: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44289/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":314,"threads_started":1}
I20260810 19:19:52.127663 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 16286
I20260810 19:19:52.128818 16391 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 19:19:52.142973 16441 webserver.cc:533] Webserver: sslize failed:
E20260810 19:19:52.143119 16441 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 19:19:52.226843 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 16286
2026-08-10T19:19:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (759 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T19:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:52Z Disabled control of system clock
I20260810 19:19:52.323695 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42855
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:42855
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/testchainca.cert with env {}
W20260810 19:19:52.656721 16449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:52.657074 16449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:52.657181 16449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:52.664578 16449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:52.664690 16449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:52.664732 16449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:52.664788 16449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:52.664831 16449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 19:19:52.664897 16449 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 19:19:52.664937 16449 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 19:19:52.675146 16449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:42855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:52.676442 16449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:52.677758 16449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:52.689091 16456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:52.689218 16458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:52.689702 16455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:52.690164 16449 server_base.cc:1034] running on GCE node
I20260810 19:19:52.690928 16449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:52.692339 16449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:52.693572 16449 hybrid_clock.cc:648] HybridClock initialized: now 1786389592693490 us; error 49 us; skew 500 ppm
I20260810 19:19:52.693903 16449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:19:52.696162 16449 webserver.cc:460] Webserver started at http://127.7.43.62:42103/ using document root <none> and password file <none>
I20260810 19:19:52.696851 16449 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:52.696956 16449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:52.697269 16449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:52.700209 16449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "be2c134da703402daa46051bc5bfbcd2"
format_stamp: "Formatted at 2026-08-10 19:19:52 on dist-test-slave-7lbf"
server_key: "236b2a4821479638549c71490ddf09e7"
server_key_iv: "deeba6b4201254405fb29cbd5a5218b0"
server_key_version: "encryptionkey@0"
I20260810 19:19:52.700992 16449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "be2c134da703402daa46051bc5bfbcd2"
format_stamp: "Formatted at 2026-08-10 19:19:52 on dist-test-slave-7lbf"
server_key: "236b2a4821479638549c71490ddf09e7"
server_key_iv: "deeba6b4201254405fb29cbd5a5218b0"
server_key_version: "encryptionkey@0"
I20260810 19:19:52.706027 16449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 19:19:52.709352 16467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:52.710717 16449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:52.710886 16449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "be2c134da703402daa46051bc5bfbcd2"
format_stamp: "Formatted at 2026-08-10 19:19:52 on dist-test-slave-7lbf"
server_key: "236b2a4821479638549c71490ddf09e7"
server_key_iv: "deeba6b4201254405fb29cbd5a5218b0"
server_key_version: "encryptionkey@0"
I20260810 19:19:52.711092 16449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:52.732201 16449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:52.733064 16449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:52.733345 16449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:52.750726 16449 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:42855
I20260810 19:19:52.750748 16553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:42855 every 8 connection(s)
I20260810 19:19:52.752050 16449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:52.755731 16554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:52.756312 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 16449
I20260810 19:19:52.756537 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:52.757164 7340 external_mini_cluster.cc:1428] Setting key 094100620b6dbc127eb65b6327f523cd
I20260810 19:19:52.770422 16554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2: Bootstrap starting.
I20260810 19:19:52.772310 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:52.759126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.7.43.62:42855)
0810 19:19:52.760215 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:52.760229 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:52.760288 (+ 59us) server_negotiation.cc:407] Connection header received
0810 19:19:52.762242 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:52.762272 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:52.762903 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:52.763351 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:52.763876 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.765751 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.767093 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.768189 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.768355 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:52.769484 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:52.769495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:52.769536 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:19:52.770238 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:52.770339 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:19:52.770365 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:52.770501 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:52.770714 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 19:19:52.771042 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":343,"threads_started":1}
I20260810 19:19:52.774103 16554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:52.775209 16554 log.cc:824] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:52.776105 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:19:52.778026 16554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2: No bootstrap required, opened a new log
I20260810 19:19:52.785893 16554 raft_consensus.cc:348] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2c134da703402daa46051bc5bfbcd2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42855 } }
I20260810 19:19:52.786334 16554 raft_consensus.cc:374] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:52.786453 16554 raft_consensus.cc:729] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2c134da703402daa46051bc5bfbcd2, State: Initialized, Role: FOLLOWER
I20260810 19:19:52.787058 16554 consensus_queue.cc:260] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2c134da703402daa46051bc5bfbcd2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42855 } }
I20260810 19:19:52.787333 16554 raft_consensus.cc:388] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:52.787497 16554 raft_consensus.cc:482] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:52.787678 16554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:52.789973 16554 raft_consensus.cc:504] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2c134da703402daa46051bc5bfbcd2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42855 } }
I20260810 19:19:52.790526 16554 leader_election.cc:302] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be2c134da703402daa46051bc5bfbcd2; no voters:
I20260810 19:19:52.791682 16554 leader_election.cc:288] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:52.791795 16562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:52.794174 16554 sys_catalog.cc:564] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:52.794575 16562 raft_consensus.cc:686] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [term 1 LEADER]: Becoming Leader. State: Replica: be2c134da703402daa46051bc5bfbcd2, State: Running, Role: LEADER
I20260810 19:19:52.795094 16562 consensus_queue.cc:237] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2c134da703402daa46051bc5bfbcd2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42855 } }
I20260810 19:19:52.797940 16566 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be2c134da703402daa46051bc5bfbcd2. Latest consensus state: current_term: 1 leader_uuid: "be2c134da703402daa46051bc5bfbcd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2c134da703402daa46051bc5bfbcd2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42855 } } }
I20260810 19:19:52.798344 16566 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:52.798622 16564 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be2c134da703402daa46051bc5bfbcd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2c134da703402daa46051bc5bfbcd2" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42855 } } }
I20260810 19:19:52.799055 16564 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:52.801795 16585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:52.807665 16585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:52.816376 16585 catalog_manager.cc:1347] Generated new cluster ID: cea9e4fb306549ed82c0b0d9b77c8241
I20260810 19:19:52.816469 16585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:52.846352 16585 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:52.847611 16585 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:52.863517 16585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be2c134da703402daa46051bc5bfbcd2: Generated new TSK 0
I20260810 19:19:52.864349 16585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:19:52.932119 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:52.876289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.7.43.62:42855)
0810 19:19:52.876475 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:52.876482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:52.876884 (+ 402us) server_negotiation.cc:407] Connection header received
0810 19:19:52.877095 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:52.877099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:52.877171 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:52.877309 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0810 19:19:52.877837 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.878560 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.879938 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.880418 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.880533 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:52.881030 (+ 497us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 19:19:52.931353 (+ 50323us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 19:19:52.931785 (+ 432us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53794: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 19:19:52.932279 16559 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53794: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 19:19:52.932360 16600 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:52.876213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42855 (local address 127.0.0.1:53794)
0810 19:19:52.876778 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:52.876793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:52.876933 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:52.877336 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:52.877341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:52.877353 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0810 19:19:52.877687 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:52.877697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.878710 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.878713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:52.879799 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:52.879810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:52.880534 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:52.880595 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:52.880740 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:52.880774 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 19:19:52.931662 (+ 50888us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 19:19:52.931964 (+ 302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.43.62:42855: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":313,"threads_started":1}
I20260810 19:19:52.933734 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 16449
I20260810 19:19:52.935436 16548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:53.034438 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 16449
2026-08-10T19:19:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (788 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (417 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (410 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:53 dist-test-slave-7lbf krb5kdc[16621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:53 dist-test-slave-7lbf krb5kdc[16621](info): set up 2 sockets
Aug 10 19:19:53 dist-test-slave-7lbf krb5kdc[16621](info): commencing operation
krb5kdc: starting...
W20260810 19:19:55.091096 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:55 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:19:55.344838 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40493
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:55.672484 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:55.672823 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:55.672972 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:55.680539 16646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:55.680678 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:55.680759 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:55.680843 16646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:55.680902 16646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:55.691160 16646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:55.692525 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:55.693890 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:55.705993 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:55.706055 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:55.706120 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:55.706880 16646 server_base.cc:1034] running on GCE node
I20260810 19:19:55.707701 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:55.709185 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:55.710393 16646 hybrid_clock.cc:648] HybridClock initialized: now 1786389595710336 us; error 46 us; skew 500 ppm
I20260810 19:19:55.710777 16646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:55 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389595, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:55.713886 16646 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:19:55.715418 16646 webserver.cc:460] Webserver started at http://127.7.43.62:40501/ using document root <none> and password file <none>
I20260810 19:19:55.716151 16646 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:55.716284 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:55.716609 16646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:55.719671 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "85e0e8216e9c481a8333ca855fbaac3c"
format_stamp: "Formatted at 2026-08-10 19:19:55 on dist-test-slave-7lbf"
server_key: "5cb595a9240abd49178270a12c946d04"
server_key_iv: "593bab4502312cffe5f58fa01a853376"
server_key_version: "encryptionkey@0"
I20260810 19:19:55.720459 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "85e0e8216e9c481a8333ca855fbaac3c"
format_stamp: "Formatted at 2026-08-10 19:19:55 on dist-test-slave-7lbf"
server_key: "5cb595a9240abd49178270a12c946d04"
server_key_iv: "593bab4502312cffe5f58fa01a853376"
server_key_version: "encryptionkey@0"
I20260810 19:19:55.725977 16646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:19:55.729494 16670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:55.730867 16646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:55.731035 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "85e0e8216e9c481a8333ca855fbaac3c"
format_stamp: "Formatted at 2026-08-10 19:19:55 on dist-test-slave-7lbf"
server_key: "5cb595a9240abd49178270a12c946d04"
server_key_iv: "593bab4502312cffe5f58fa01a853376"
server_key_version: "encryptionkey@0"
I20260810 19:19:55.731215 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:55.755312 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:55.759439 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:55.759716 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:55.777486 16646 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:40493
I20260810 19:19:55.777506 16753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:40493 every 8 connection(s)
I20260810 19:19:55.778923 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:55.782756 16755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:19:55.787684 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 16646
I20260810 19:19:55.787901 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:55.788558 7340 external_mini_cluster.cc:1428] Setting key 769fbf830e2097633da85a8b06be472e
I20260810 19:19:55.794832 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c: Bootstrap starting.
I20260810 19:19:55.798745 16755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:55.799821 16755 log.cc:824] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:55.802856 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c: No bootstrap required, opened a new log
Aug 10 19:19:55 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:55.813059 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:55.790643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.7.43.62:40493)
0810 19:19:55.791544 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:55.791561 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:55.791625 (+ 64us) server_negotiation.cc:407] Connection header received
0810 19:19:55.794027 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:55.794058 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:55.794571 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:55.795049 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:55.796300 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:55.798347 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:55.799596 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:55.800630 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:55.800766 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:55.804648 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:55.804662 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:55.804681 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:55.804723 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:55.808673 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:55.809292 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:55.809300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:55.809310 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:55.809552 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:55.809888 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:55.809892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:55.809896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:55.810618 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:55.810904 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:55.811229 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 19:19:55.811685 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":388,"threads_started":1}
I20260810 19:19:55.812938 16755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0e8216e9c481a8333ca855fbaac3c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40493 } }
I20260810 19:19:55.813452 16755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:55.813607 16755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e0e8216e9c481a8333ca855fbaac3c, State: Initialized, Role: FOLLOWER
I20260810 19:19:55.814198 16755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0e8216e9c481a8333ca855fbaac3c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40493 } }
I20260810 19:19:55.814426 16755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:55.814574 16755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:55.814778 16755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:55.817337 16755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0e8216e9c481a8333ca855fbaac3c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40493 } }
I20260810 19:19:55.817823 16755 leader_election.cc:302] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e0e8216e9c481a8333ca855fbaac3c; no voters:
I20260810 19:19:55.818361 16755 leader_election.cc:288] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:55.818526 16761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:55.819150 16761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [term 1 LEADER]: Becoming Leader. State: Replica: 85e0e8216e9c481a8333ca855fbaac3c, State: Running, Role: LEADER
I20260810 19:19:55.819746 16761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0e8216e9c481a8333ca855fbaac3c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40493 } }
I20260810 19:19:55.820303 16755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:55.822854 16763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e0e8216e9c481a8333ca855fbaac3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0e8216e9c481a8333ca855fbaac3c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40493 } } }
I20260810 19:19:55.822830 16765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e0e8216e9c481a8333ca855fbaac3c. Latest consensus state: current_term: 1 leader_uuid: "85e0e8216e9c481a8333ca855fbaac3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0e8216e9c481a8333ca855fbaac3c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40493 } } }
I20260810 19:19:55.823283 16763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:55.823283 16765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:55.829097 16774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:55.833870 16774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:55.842341 16774 catalog_manager.cc:1347] Generated new cluster ID: 39486fd992014d78b70d09bcc4cac5fe
I20260810 19:19:55.842461 16774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:19:55.852849 16774 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:19:55.854206 16774 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:19:55.866060 16774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e0e8216e9c481a8333ca855fbaac3c: Generated new TSK 0
I20260810 19:19:55.867033 16774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:19:55.936972 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40493
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:56.270432 16793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:56.270730 16793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:56.270838 16793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:56.278038 16793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:56.278143 16793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:56.278262 16793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:19:56.288146 16793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:56.289382 16793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:56.290823 16793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:56.304068 16801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:56.304030 16802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:56.304065 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:56.304929 16793 server_base.cc:1034] running on GCE node
I20260810 19:19:56.305600 16793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:56.306640 16793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:56.307898 16793 hybrid_clock.cc:648] HybridClock initialized: now 1786389596307839 us; error 43 us; skew 500 ppm
I20260810 19:19:56.308229 16793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:56 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389596, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:56.314369 16793 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:19:56.315809 16793 webserver.cc:460] Webserver started at http://127.7.43.1:44839/ using document root <none> and password file <none>
I20260810 19:19:56.316493 16793 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:56.316612 16793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:56.316906 16793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:56.319895 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "f2e3a187c5df4e06a2663251d956b319"
format_stamp: "Formatted at 2026-08-10 19:19:56 on dist-test-slave-7lbf"
server_key: "735eed5672c4288014eef3d2620d6b27"
server_key_iv: "523cc213c1878ae629b6cb9ef00f9f37"
server_key_version: "encryptionkey@0"
I20260810 19:19:56.320708 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e3a187c5df4e06a2663251d956b319"
format_stamp: "Formatted at 2026-08-10 19:19:56 on dist-test-slave-7lbf"
server_key: "735eed5672c4288014eef3d2620d6b27"
server_key_iv: "523cc213c1878ae629b6cb9ef00f9f37"
server_key_version: "encryptionkey@0"
I20260810 19:19:56.325923 16793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:19:56.329542 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:56.330924 16793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:56.331097 16793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "f2e3a187c5df4e06a2663251d956b319"
format_stamp: "Formatted at 2026-08-10 19:19:56 on dist-test-slave-7lbf"
server_key: "735eed5672c4288014eef3d2620d6b27"
server_key_iv: "523cc213c1878ae629b6cb9ef00f9f37"
server_key_version: "encryptionkey@0"
I20260810 19:19:56.331296 16793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:56.354640 16793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:56.358961 16793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:56.359246 16793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:56.360219 16793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:56.361617 16793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:56.361698 16793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:56.361824 16793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:56.361878 16793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:56.392764 16793 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:33005
I20260810 19:19:56.392834 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:33005 every 8 connection(s)
I20260810 19:19:56.394083 16793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:19:56.402040 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 16793
I20260810 19:19:56.402279 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:19:56.403045 7340 external_mini_cluster.cc:1428] Setting key 5974c77c58ee02aa3ec4d9f84827410d
Aug 10 19:19:56 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389596, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:56.414500 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:56.396637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:41807 (local address 127.7.43.62:40493)
0810 19:19:56.397068 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:56.397072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:56.398101 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 19:19:56.399856 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:56.399861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:56.399931 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:56.400028 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:56.401479 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.402597 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.404283 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.404867 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.405018 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:56.408915 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:56.408921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:56.408924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:56.408962 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:56.411469 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:56.412282 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:56.412286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:56.412288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:56.412401 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:56.412822 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:56.412825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:56.412827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:56.413079 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:56.413188 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:56.414056 (+ 868us) server_negotiation.cc:299] Negotiation successful
0810 19:19:56.414229 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":207,"threads_started":1}
I20260810 19:19:56.415807 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:56.397172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40493 (local address 127.7.43.1:41807)
0810 19:19:56.397935 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:56.397977 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:56.399607 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:56.400227 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:56.400235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:56.400745 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:56.401309 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:56.401326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.402753 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.402757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:56.404147 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:56.404159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.405104 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.405112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:56.405390 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:56.406081 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:56.406109 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:56.408695 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:56.411605 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:56.411613 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:56.411630 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:56.412185 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:56.412492 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:56.412496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:56.412499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:56.412734 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:56.413197 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:56.413205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:56.413651 (+ 446us) client_negotiation.cc:769] Sending connection context
0810 19:19:56.413973 (+ 322us) client_negotiation.cc:240] Negotiation successful
0810 19:19:56.414490 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":315,"threads_started":1}
I20260810 19:19:56.417447 16985 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40493
I20260810 19:19:56.417783 16985 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:56.418509 16985 heartbeater.cc:511] Master 127.7.43.62:40493 requested a full tablet report, sending...
I20260810 19:19:56.420608 16693 ts_manager.cc:194] Registered new tserver with Master: f2e3a187c5df4e06a2663251d956b319 (127.7.43.1:33005)
I20260810 19:19:56.422199 16693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:41807
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:19:56.460664 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40493
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:56.790614 16998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:56.790997 16998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:56.791159 16998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:56.799065 16998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:56.799165 16998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:56.799312 16998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:19:56.809037 16998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:56.810317 16998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:56.811558 16998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:56.824057 17010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:56.824133 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:56.824110 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:56.824694 16998 server_base.cc:1034] running on GCE node
I20260810 19:19:56.825225 16998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:56.826116 16998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:56.827294 16998 hybrid_clock.cc:648] HybridClock initialized: now 1786389596827239 us; error 48 us; skew 500 ppm
I20260810 19:19:56.827575 16998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:56 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389596, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:56.830700 16998 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:19:56.832034 16998 webserver.cc:460] Webserver started at http://127.7.43.2:33233/ using document root <none> and password file <none>
I20260810 19:19:56.832652 16998 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:56.832749 16998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:56.833012 16998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:56.836031 16998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "9bca1e61055e4a94b769c6ba5630714a"
format_stamp: "Formatted at 2026-08-10 19:19:56 on dist-test-slave-7lbf"
server_key: "b45f58d554d7b3171270f79ad3e8a14c"
server_key_iv: "46b13792b46845d83bdb589fca3b4898"
server_key_version: "encryptionkey@0"
I20260810 19:19:56.836789 16998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "9bca1e61055e4a94b769c6ba5630714a"
format_stamp: "Formatted at 2026-08-10 19:19:56 on dist-test-slave-7lbf"
server_key: "b45f58d554d7b3171270f79ad3e8a14c"
server_key_iv: "46b13792b46845d83bdb589fca3b4898"
server_key_version: "encryptionkey@0"
I20260810 19:19:56.841616 16998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:19:56.844801 17020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:56.846108 16998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:19:56.846251 16998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "9bca1e61055e4a94b769c6ba5630714a"
format_stamp: "Formatted at 2026-08-10 19:19:56 on dist-test-slave-7lbf"
server_key: "b45f58d554d7b3171270f79ad3e8a14c"
server_key_iv: "46b13792b46845d83bdb589fca3b4898"
server_key_version: "encryptionkey@0"
I20260810 19:19:56.846443 16998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:56.862910 16998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:56.866627 16998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:56.866878 16998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:56.867774 16998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:56.869086 16998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:56.869158 16998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:56.869287 16998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:56.869338 16998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:56.899572 16998 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:33821
I20260810 19:19:56.899610 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:33821 every 8 connection(s)
I20260810 19:19:56.900918 16998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:19:56.903932 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 16998
I20260810 19:19:56.904143 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:19:56.904750 7340 external_mini_cluster.cc:1428] Setting key 9e7572ff7efd993d385addb0f9c28b66
Aug 10 19:19:56 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389596, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:56.920625 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:56.903557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:44119 (local address 127.7.43.62:40493)
0810 19:19:56.903740 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:56.903744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:56.905198 (+ 1454us) server_negotiation.cc:407] Connection header received
0810 19:19:56.907210 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:56.907216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:56.907296 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:56.907381 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:56.908900 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.909645 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.911214 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.911625 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.911736 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:56.915122 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:56.915127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:56.915130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:56.915162 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:56.917701 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:56.918412 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:56.918417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:56.918421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:56.918574 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:56.918969 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:56.918973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:56.918976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:56.919291 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:56.919422 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:56.920085 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 19:19:56.920301 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:19:56.921564 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:56.904192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40493 (local address 127.7.43.2:44119)
0810 19:19:56.905032 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:56.905076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:56.906944 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:56.907639 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:56.907648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:56.908164 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:56.908739 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:56.908756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.909792 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.909796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:56.911062 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:56.911075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:56.911845 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:56.911853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:56.912077 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:56.912699 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:56.912721 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:56.914904 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:56.917821 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:56.917827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:56.917840 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:56.918290 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:56.918681 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:56.918684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:56.918686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:56.918868 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:56.919396 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:56.919402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:56.919714 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 19:19:56.919942 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 19:19:56.920344 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":337,"threads_started":1}
I20260810 19:19:56.923031 17217 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40493
I20260810 19:19:56.923318 17217 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:56.924046 17217 heartbeater.cc:511] Master 127.7.43.62:40493 requested a full tablet report, sending...
I20260810 19:19:56.925576 16693 ts_manager.cc:194] Registered new tserver with Master: 9bca1e61055e4a94b769c6ba5630714a (127.7.43.2:33821)
I20260810 19:19:56.926676 16693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:44119
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:19:56.960887 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40493
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:57.288686 17222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:57.288990 17222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:57.289096 17222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:57.296324 17222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:57.296430 17222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:57.296550 17222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:19:57.306463 17222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:57.307686 17222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:57.308910 17222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:57.321343 17233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:57.321404 17230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:57.321408 17231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:57.322173 17222 server_base.cc:1034] running on GCE node
I20260810 19:19:57.322749 17222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:57.323657 17222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:57.324865 17222 hybrid_clock.cc:648] HybridClock initialized: now 1786389597324803 us; error 52 us; skew 500 ppm
I20260810 19:19:57.325160 17222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.328145 17222 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:19:57.329592 17222 webserver.cc:460] Webserver started at http://127.7.43.3:38475/ using document root <none> and password file <none>
I20260810 19:19:57.330289 17222 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:57.330392 17222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:57.330678 17222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:57.333572 17222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "d409c367452c43ad801c7d48c7c772f5"
format_stamp: "Formatted at 2026-08-10 19:19:57 on dist-test-slave-7lbf"
server_key: "d380901456fa2e1443f6a877032852cb"
server_key_iv: "bcb484d96b7063ec192bcadde69e3cfd"
server_key_version: "encryptionkey@0"
I20260810 19:19:57.334354 17222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "d409c367452c43ad801c7d48c7c772f5"
format_stamp: "Formatted at 2026-08-10 19:19:57 on dist-test-slave-7lbf"
server_key: "d380901456fa2e1443f6a877032852cb"
server_key_iv: "bcb484d96b7063ec192bcadde69e3cfd"
server_key_version: "encryptionkey@0"
I20260810 19:19:57.339505 17222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 19:19:57.342978 17244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:57.344354 17222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:19:57.344512 17222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "d409c367452c43ad801c7d48c7c772f5"
format_stamp: "Formatted at 2026-08-10 19:19:57 on dist-test-slave-7lbf"
server_key: "d380901456fa2e1443f6a877032852cb"
server_key_iv: "bcb484d96b7063ec192bcadde69e3cfd"
server_key_version: "encryptionkey@0"
I20260810 19:19:57.344696 17222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:57.365964 17222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:57.370107 17222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:57.370400 17222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:57.371419 17222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:19:57.372778 17222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:19:57.372862 17222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:57.372985 17222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:19:57.373041 17222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:57.404203 17222 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:42571
I20260810 19:19:57.404263 17423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:42571 every 8 connection(s)
I20260810 19:19:57.405449 17222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:19:57.414129 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 17222
I20260810 19:19:57.414312 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:19:57.414770 7340 external_mini_cluster.cc:1428] Setting key f9aaba3e7cd0043e69dc825d290278e1
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:19:57.425176 16985 heartbeater.cc:503] Master 127.7.43.62:40493 was elected leader, sending a full tablet report...
I20260810 19:19:57.425732 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.408034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:41401 (local address 127.7.43.62:40493)
0810 19:19:57.408224 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.408228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.409425 (+ 1197us) server_negotiation.cc:407] Connection header received
0810 19:19:57.411288 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.411293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.411366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.411457 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.412945 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.413787 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.415429 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.415960 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.416079 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.419865 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.419871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.419874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.419921 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.422705 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.423407 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.423411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.423414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.423547 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.423923 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.423928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.423930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.424258 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.424384 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.425134 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.425387 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:19:57.426647 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.408512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40493 (local address 127.7.43.3:41401)
0810 19:19:57.409252 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 19:19:57.409298 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:19:57.411004 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:19:57.411667 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:19:57.411675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:19:57.412193 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:19:57.412773 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:57.412790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.413917 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.413921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:57.415260 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:19:57.415278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.416182 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.416189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:19:57.416389 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.417119 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:19:57.417150 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:19:57.419615 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:19:57.422844 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:57.422852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:57.422866 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:57.423303 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:57.423627 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:19:57.423630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:19:57.423632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:19:57.423809 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:19:57.424362 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:19:57.424371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:19:57.424711 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 19:19:57.424970 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 19:19:57.425412 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":288,"threads_started":1}
I20260810 19:19:57.428025 17425 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40493
I20260810 19:19:57.428431 17425 heartbeater.cc:464] Registering TS with master...
I20260810 19:19:57.429386 17425 heartbeater.cc:511] Master 127.7.43.62:40493 requested a full tablet report, sending...
I20260810 19:19:57.430825 16693 ts_manager.cc:194] Registered new tserver with Master: d409c367452c43ad801c7d48c7c772f5 (127.7.43.3:42571)
I20260810 19:19:57.431600 16693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:41401
I20260810 19:19:57.442512 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/test-admin.keytab.
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.475425 7340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 19:19:57.480705 17437 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:57.478207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.7.43.1:33005)
0810 19:19:57.478977 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.478987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.479016 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:19:57.479086 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.479093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.479303 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.479447 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.480340 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49822: BlockingRecv error: recv got EOF from 127.0.0.1:49822 (error 108)
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":410,"threads_started":1}
W20260810 19:19:57.480818 7340 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc' of type FILE
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.482486 7340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.483953 7340 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:57.499477 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.485562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.7.43.1:33005)
0810 19:19:57.485732 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.485738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.486005 (+ 267us) server_negotiation.cc:407] Connection header received
0810 19:19:57.486165 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.486169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.486229 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.486314 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.487399 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.488367 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.489594 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.490162 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.490279 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.493779 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.493791 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.493807 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.493858 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.496608 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.497339 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.497346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.497348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.497499 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.497884 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.497887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.497889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.498265 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.498568 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.498910 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.499180 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 19:19:57.505306 17437 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:57.503677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.7.43.1:33005)
0810 19:19:57.503858 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.503863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.504169 (+ 306us) server_negotiation.cc:407] Connection header received
0810 19:19:57.504258 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.504262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.504319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.504395 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.505052 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49844: BlockingRecv error: recv got EOF from 127.0.0.1:49844 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 19:19:57.505873 7340 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc)
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.507392 7340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.508759 7340 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:57.522604 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.510249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.7.43.1:33005)
0810 19:19:57.510458 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.510464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.510583 (+ 119us) server_negotiation.cc:407] Connection header received
0810 19:19:57.510747 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.510750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.510806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.510885 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.511836 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.512568 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.513833 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.514300 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.514438 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.518024 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.518029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.518032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.518067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.520390 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.521010 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.521014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.521018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.521165 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.521494 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.521497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.521499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.521791 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.521984 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.522200 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.522360 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 19:19:57.527096 17437 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:19:57.525204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.7.43.1:33005)
0810 19:19:57.525370 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.525376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.525722 (+ 346us) server_negotiation.cc:407] Connection header received
0810 19:19:57.525869 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.525873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.525941 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.526019 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.526898 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49860: BlockingRecv error: recv got EOF from 127.0.0.1:49860 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 19:19:57.527318 7340 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786389541863099-7340-0/krb5kdc/krb5cc' of type FILE
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.528996 7340 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:57.530452 7340 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:19:57.544082 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.531742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.7.43.1:33005)
0810 19:19:57.531913 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.531919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.532178 (+ 259us) server_negotiation.cc:407] Connection header received
0810 19:19:57.532322 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.532327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.532393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.532479 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.533603 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.534359 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.535642 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.536056 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.536160 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.539390 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.539394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.539398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.539430 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.541672 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.542270 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.542276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.542280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.542429 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.542761 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.542767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.542769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.543038 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.543335 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.543579 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.543782 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:19:57.545487 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 16793
I20260810 19:19:57.556450 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.546150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.7.43.1:33005)
0810 19:19:57.546280 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.546284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.546655 (+ 371us) server_negotiation.cc:407] Connection header received
0810 19:19:57.546787 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.546791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.546839 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.546902 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.547833 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.548559 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.549793 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.550239 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.550331 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.552185 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.552190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.552193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.552220 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.554524 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.555055 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.555058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.555060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.555177 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.555417 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.555420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.555422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.555645 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.555824 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.556043 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.556194 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:19:57.557005 16979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:57.639305 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 16793
I20260810 19:19:57.664194 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 16998
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:19:57.678550 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.665136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.7.43.2:33821)
0810 19:19:57.665687 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.665694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.665720 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:19:57.665788 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.665795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.666012 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.666140 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.667042 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.668041 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.669173 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.669758 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.669853 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.673376 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.673388 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.673402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.673435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.676163 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.676726 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.676733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.676735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.676851 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.677128 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.677131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.677133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.677453 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.677711 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.677997 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.678237 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":260,"threads_started":1}
I20260810 19:19:57.679795 17208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:57.760680 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 16998
I20260810 19:19:57.777460 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 17222
Aug 10 19:19:57 dist-test-slave-7lbf krb5kdc[16621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:19:57.793800 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:57.778526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.7.43.3:42571)
0810 19:19:57.779234 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:57.779241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:57.779273 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:19:57.779376 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:57.779386 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:57.779624 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:57.779752 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:57.781063 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.782164 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.783895 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:57.784451 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:57.784543 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:57.788678 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:57.788691 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:57.788707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:57.788746 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:57.791418 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.792069 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.792075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.792078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.792194 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:57.792488 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:57.792491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:57.792493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:57.792813 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:57.793041 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:57.793252 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 19:19:57.793480 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":381,"threads_started":1}
I20260810 19:19:57.795185 17419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:57.875655 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 17222
I20260810 19:19:57.893651 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 16646
I20260810 19:19:57.894865 16749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:19:57.997020 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 16646
2026-08-10T19:19:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4139 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:19:58 dist-test-slave-7lbf krb5kdc[17469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:19:58 dist-test-slave-7lbf krb5kdc[17469](info): set up 2 sockets
Aug 10 19:19:58 dist-test-slave-7lbf krb5kdc[17469](info): commencing operation
krb5kdc: starting...
W20260810 19:19:59.229853 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:19:59 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:19:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab.
Entry for principal oryx/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab.
I20260810 19:19:59.491528 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38017
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab
--principal=oryx/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:19:59.829970 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:19:59.830288 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:19:59.830425 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:19:59.837641 17491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:19:59.837757 17491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:19:59.837837 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:19:59.837920 17491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:19:59.837984 17491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:19:59.847941 17491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab
--principal=oryx/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:19:59.849274 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:19:59.850642 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:19:59.862303 17500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:59.862336 17505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:19:59.862241 17501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:19:59.862998 17491 server_base.cc:1034] running on GCE node
I20260810 19:19:59.863754 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:19:59.865190 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:19:59.866380 17491 hybrid_clock.cc:648] HybridClock initialized: now 1786389599866320 us; error 46 us; skew 500 ppm
I20260810 19:19:59.866709 17491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:19:59 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:19:59.869781 17491 init.cc:377] Logged in from keytab as oryx/127.7.43.62@KRBTEST.COM (short username oryx)
I20260810 19:19:59.871199 17491 webserver.cc:460] Webserver started at http://127.7.43.62:40643/ using document root <none> and password file <none>
I20260810 19:19:59.871850 17491 fs_manager.cc:359] Metadata directory not provided
I20260810 19:19:59.871950 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:19:59.872210 17491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:19:59.875306 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "83bb9449eafe4dcd909a5eaf6160fbbf"
format_stamp: "Formatted at 2026-08-10 19:19:59 on dist-test-slave-7lbf"
server_key: "ab64c0defcc55af60f6363f724d87229"
server_key_iv: "b646d7324feb434652ec131f64df1961"
server_key_version: "encryptionkey@0"
I20260810 19:19:59.876036 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "83bb9449eafe4dcd909a5eaf6160fbbf"
format_stamp: "Formatted at 2026-08-10 19:19:59 on dist-test-slave-7lbf"
server_key: "ab64c0defcc55af60f6363f724d87229"
server_key_iv: "b646d7324feb434652ec131f64df1961"
server_key_version: "encryptionkey@0"
I20260810 19:19:59.881197 17491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:19:59.884652 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:19:59.885998 17491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 19:19:59.886142 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "83bb9449eafe4dcd909a5eaf6160fbbf"
format_stamp: "Formatted at 2026-08-10 19:19:59 on dist-test-slave-7lbf"
server_key: "ab64c0defcc55af60f6363f724d87229"
server_key_iv: "b646d7324feb434652ec131f64df1961"
server_key_version: "encryptionkey@0"
I20260810 19:19:59.886305 17491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:19:59.901741 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:19:59.905973 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:19:59.906211 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:19:59.923245 17491 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:38017
I20260810 19:19:59.923269 17599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:38017 every 8 connection(s)
I20260810 19:19:59.924640 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:19:59.925709 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 17491
I20260810 19:19:59.925962 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:19:59.926750 7340 external_mini_cluster.cc:1428] Setting key 814eeaf4d6ef70dc254949dd0ef25803
I20260810 19:19:59.929477 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:19:59 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:19:59.942408 17600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Bootstrap starting.
I20260810 19:19:59.945779 17600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Neither blocks nor log segments found. Creating new log.
I20260810 19:19:59.947078 17600 log.cc:824] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:19:59.949764 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:19:59.928846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.7.43.62:38017)
0810 19:19:59.929638 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0810 19:19:59.929653 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:19:59.929703 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:19:59.931902 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:19:59.931932 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:19:59.932427 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:19:59.932880 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:19:59.934202 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:59.936223 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:59.937407 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:19:59.938409 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:19:59.938583 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:19:59.942659 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:19:59.942667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:19:59.942681 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:19:59.942721 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:19:59.945664 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:59.946462 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:59.946469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:59.946475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:59.946634 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:19:59.947047 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:19:59.947050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:19:59.947052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:19:59.947613 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:19:59.947918 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 19:19:59.948359 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 19:19:59.948766 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":311,"threads_started":1}
I20260810 19:19:59.950007 17600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: No bootstrap required, opened a new log
I20260810 19:19:59.958051 17600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } }
I20260810 19:19:59.958456 17600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:19:59.958559 17600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bb9449eafe4dcd909a5eaf6160fbbf, State: Initialized, Role: FOLLOWER
I20260810 19:19:59.959100 17600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } }
I20260810 19:19:59.959326 17600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:19:59.959457 17600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:19:59.959615 17600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:19:59.962070 17600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } }
I20260810 19:19:59.962628 17600 leader_election.cc:302] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83bb9449eafe4dcd909a5eaf6160fbbf; no voters:
I20260810 19:19:59.963181 17600 leader_election.cc:288] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:19:59.963304 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:19:59.963982 17607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 LEADER]: Becoming Leader. State: Replica: 83bb9449eafe4dcd909a5eaf6160fbbf, State: Running, Role: LEADER
I20260810 19:19:59.964526 17607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } }
I20260810 19:19:59.965247 17600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:19:59.967187 17610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83bb9449eafe4dcd909a5eaf6160fbbf. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } }
I20260810 19:19:59.967161 17608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } }
I20260810 19:19:59.967695 17608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:59.969550 17610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:19:59.970343 17613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:19:59.975467 17613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:19:59.984738 17613 catalog_manager.cc:1347] Generated new cluster ID: 84754b7befde449a98e32871b6cfdf53
I20260810 19:19:59.984859 17613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:00.001612 17613 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:20:00.003563 17613 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:00.012650 17613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Generated new TSK 0
I20260810 19:20:00.013871 17613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab.
Entry for principal oryx/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab.
I20260810 19:20:00.077008 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38017
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:00.434368 17634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:00.434651 17634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:00.434754 17634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:00.441922 17634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:00.442036 17634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:00.442169 17634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:00.452049 17634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:00.453289 17634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:00.454622 17634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:00.468406 17644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:00.468369 17645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:00.468636 17647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:00.469233 17634 server_base.cc:1034] running on GCE node
I20260810 19:20:00.469976 17634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:00.471112 17634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:00.472339 17634 hybrid_clock.cc:648] HybridClock initialized: now 1786389600472280 us; error 31 us; skew 500 ppm
I20260810 19:20:00.472671 17634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:00 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389600, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:00.475916 17634 init.cc:377] Logged in from keytab as oryx/127.7.43.1@KRBTEST.COM (short username oryx)
I20260810 19:20:00.477332 17634 webserver.cc:460] Webserver started at http://127.7.43.1:46775/ using document root <none> and password file <none>
I20260810 19:20:00.478055 17634 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:00.478165 17634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:00.478494 17634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:00.481585 17634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "c64aaeddbe424cea978f4db16dfd3778"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "8f410f0145661b99cd4391696ffcbe25"
server_key_iv: "bef443d5225d4340150e48af963384fd"
server_key_version: "encryptionkey@0"
I20260810 19:20:00.482358 17634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "c64aaeddbe424cea978f4db16dfd3778"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "8f410f0145661b99cd4391696ffcbe25"
server_key_iv: "bef443d5225d4340150e48af963384fd"
server_key_version: "encryptionkey@0"
I20260810 19:20:00.487604 17634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:20:00.491312 17657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:00.492655 17634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 19:20:00.492820 17634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "c64aaeddbe424cea978f4db16dfd3778"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "8f410f0145661b99cd4391696ffcbe25"
server_key_iv: "bef443d5225d4340150e48af963384fd"
server_key_version: "encryptionkey@0"
I20260810 19:20:00.493023 17634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:00.508994 17634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:00.513366 17634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:00.513675 17634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:00.514664 17634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:00.516049 17634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:00.516146 17634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:00.516273 17634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:00.516335 17634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:00.546492 17634 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:36603
I20260810 19:20:00.546531 17849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:36603 every 8 connection(s)
I20260810 19:20:00.547796 17634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:00.548126 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 17634
I20260810 19:20:00.548332 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:20:00.549175 7340 external_mini_cluster.cc:1428] Setting key a56b252b6f4c31b3e769bb4345d6940f
Aug 10 19:20:00 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389600, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:00.567617 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:00.550561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:43983 (local address 127.7.43.62:38017)
0810 19:20:00.551232 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:00.551237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:00.552223 (+ 986us) server_negotiation.cc:407] Connection header received
0810 19:20:00.554115 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:00.554121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:00.554202 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:00.554296 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:00.555833 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:00.556545 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:00.558164 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:00.558562 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:00.558678 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:00.562054 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:00.562058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:00.562062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:00.562093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:00.564477 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:00.565206 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:00.565212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:00.565215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:00.565371 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:00.565831 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:00.565836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:00.565839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:00.566170 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:00.566308 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:00.567030 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 19:20:00.567286 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":252,"threads_started":1}
I20260810 19:20:00.568329 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:00.551183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:43983)
0810 19:20:00.552056 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:00.552095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:00.553841 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:00.554565 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:00.554574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:00.555093 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:00.555661 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:00.555679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:00.556702 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:00.556705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:00.558017 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:00.558029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:00.558793 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:00.558801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:00.559028 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:00.559645 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:00.559672 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:00.561842 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:00.564605 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:00.564614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:00.564631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:00.565041 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:00.565471 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:00.565474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:00.565476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:00.565709 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:00.566290 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:00.566296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:00.566618 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 19:20:00.566864 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 19:20:00.567245 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":404,"threads_started":1}
I20260810 19:20:00.569837 17851 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38017
I20260810 19:20:00.570220 17851 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:00.571120 17851 heartbeater.cc:511] Master 127.7.43.62:38017 requested a full tablet report, sending...
I20260810 19:20:00.573400 17543 ts_manager.cc:194] Registered new tserver with Master: c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603)
I20260810 19:20:00.574944 17543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:43983
WARNING: no policy specified for oryx/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab.
Entry for principal oryx/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab.
I20260810 19:20:00.608417 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38017
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:00.947896 17862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:00.948197 17862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:00.948334 17862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:00.955554 17862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:00.955673 17862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:00.955821 17862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:00.965399 17862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:00.966727 17862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:00.968053 17862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:00.981175 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:00.981208 17876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:00.981103 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:00.982270 17862 server_base.cc:1034] running on GCE node
I20260810 19:20:00.982854 17862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:00.983783 17862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:00.984984 17862 hybrid_clock.cc:648] HybridClock initialized: now 1786389600984932 us; error 45 us; skew 500 ppm
I20260810 19:20:00.985339 17862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:00 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389600, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:00.988502 17862 init.cc:377] Logged in from keytab as oryx/127.7.43.2@KRBTEST.COM (short username oryx)
I20260810 19:20:00.990118 17862 webserver.cc:460] Webserver started at http://127.7.43.2:40483/ using document root <none> and password file <none>
I20260810 19:20:00.990957 17862 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:00.991081 17862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:00.991395 17862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:00.994483 17862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "fb44bebcde3747aca0b1d06285d1a392"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "2dbfb90013d0b9abce36e7852707f202"
server_key_iv: "8f4a9d74dea1e2568739fdaf0b4fdeca"
server_key_version: "encryptionkey@0"
I20260810 19:20:00.995270 17862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "fb44bebcde3747aca0b1d06285d1a392"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "2dbfb90013d0b9abce36e7852707f202"
server_key_iv: "8f4a9d74dea1e2568739fdaf0b4fdeca"
server_key_version: "encryptionkey@0"
I20260810 19:20:01.000900 17862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 19:20:01.004572 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:01.006063 17862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:20:01.006224 17862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "fb44bebcde3747aca0b1d06285d1a392"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "2dbfb90013d0b9abce36e7852707f202"
server_key_iv: "8f4a9d74dea1e2568739fdaf0b4fdeca"
server_key_version: "encryptionkey@0"
I20260810 19:20:01.006419 17862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:01.021782 17862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:01.026115 17862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:01.026405 17862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:01.027416 17862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:01.028861 17862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:01.028942 17862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:01.029057 17862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:01.029114 17862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:01.060252 17862 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:35867
I20260810 19:20:01.060297 18066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:35867 every 8 connection(s)
I20260810 19:20:01.061556 17862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:01.064256 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 17862
I20260810 19:20:01.064560 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:20:01.065131 7340 external_mini_cluster.cc:1428] Setting key 0795932a39fa9381e41ccdaf0d2dd828
Aug 10 19:20:01 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389600, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:01.084185 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:01.064206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:48725 (local address 127.7.43.62:38017)
0810 19:20:01.064430 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:01.064436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:01.065826 (+ 1390us) server_negotiation.cc:407] Connection header received
0810 19:20:01.068183 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:01.068188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:01.068259 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:01.068341 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:01.070464 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.071187 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.073311 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.073771 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.073867 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:01.077940 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:01.077946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:01.077949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:01.077982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:01.080982 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:01.081784 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:01.081789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:01.081792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:01.081951 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:01.082370 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:01.082374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:01.082377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:01.082777 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:01.082919 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:01.083620 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 19:20:01.083834 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:20:01.085040 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:01.064730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:48725)
0810 19:20:01.065614 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:01.065691 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:01.067882 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:01.068606 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:01.068617 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:01.069373 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:01.070262 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:01.070289 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.071377 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.071382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:01.073188 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:01.073203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.073990 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.073997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:01.074281 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:01.074991 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:01.075020 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:01.077694 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:01.081141 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:01.081148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:01.081163 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:01.081656 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:01.082048 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:01.082050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:01.082053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:01.082253 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:01.082882 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:01.082888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:01.083214 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 19:20:01.083484 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 19:20:01.083898 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":312,"threads_started":1}
I20260810 19:20:01.086618 18067 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38017
I20260810 19:20:01.086972 18067 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:01.087901 18067 heartbeater.cc:511] Master 127.7.43.62:38017 requested a full tablet report, sending...
I20260810 19:20:01.089628 17543 ts_manager.cc:194] Registered new tserver with Master: fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867)
I20260810 19:20:01.090513 17543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:48725
WARNING: no policy specified for oryx/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab.
Entry for principal oryx/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab.
I20260810 19:20:01.128358 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38017
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:01.470204 18073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:01.470525 18073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:01.470711 18073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:01.478302 18073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:01.478440 18073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:01.478610 18073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:01.488816 18073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:01.490283 18073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:01.491808 18073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:01.505213 18083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:01.505254 18086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:01.505184 18081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:01.506057 18073 server_base.cc:1034] running on GCE node
I20260810 19:20:01.506604 18073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:01.507503 18073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:01.508697 18073 hybrid_clock.cc:648] HybridClock initialized: now 1786389601508653 us; error 40 us; skew 500 ppm
I20260810 19:20:01.508983 18073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:01 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389601, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:01.515699 18073 init.cc:377] Logged in from keytab as oryx/127.7.43.3@KRBTEST.COM (short username oryx)
I20260810 19:20:01.517225 18073 webserver.cc:460] Webserver started at http://127.7.43.3:32909/ using document root <none> and password file <none>
I20260810 19:20:01.517997 18073 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:01.518111 18073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:01.518381 18073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:01.521387 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "bebf7e6e27244a969ae4a741d2d4093a"
format_stamp: "Formatted at 2026-08-10 19:20:01 on dist-test-slave-7lbf"
server_key: "4886a6ebe0609354f4857d9b39fbafe4"
server_key_iv: "1b817544b7d9b8f307128b42e85f603e"
server_key_version: "encryptionkey@0"
I20260810 19:20:01.522202 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "bebf7e6e27244a969ae4a741d2d4093a"
format_stamp: "Formatted at 2026-08-10 19:20:01 on dist-test-slave-7lbf"
server_key: "4886a6ebe0609354f4857d9b39fbafe4"
server_key_iv: "1b817544b7d9b8f307128b42e85f603e"
server_key_version: "encryptionkey@0"
I20260810 19:20:01.527697 18073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:20:01.531181 18093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:01.532512 18073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:20:01.532648 18073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "bebf7e6e27244a969ae4a741d2d4093a"
format_stamp: "Formatted at 2026-08-10 19:20:01 on dist-test-slave-7lbf"
server_key: "4886a6ebe0609354f4857d9b39fbafe4"
server_key_iv: "1b817544b7d9b8f307128b42e85f603e"
server_key_version: "encryptionkey@0"
I20260810 19:20:01.532820 18073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:01.567741 18073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:01.572016 18073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:01.572253 18073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:01.573163 18073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:01.574479 18073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:01.574550 18073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:01.574645 18073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:01.574687 18073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:01.577879 17851 heartbeater.cc:503] Master 127.7.43.62:38017 was elected leader, sending a full tablet report...
I20260810 19:20:01.605592 18073 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:39199
I20260810 19:20:01.605641 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:39199 every 8 connection(s)
I20260810 19:20:01.606914 18073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:01.614274 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 18073
I20260810 19:20:01.614451 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:20:01.614889 7340 external_mini_cluster.cc:1428] Setting key 62ac8cc1ca4ab97edeaf57b113d185ce
Aug 10 19:20:01 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389601, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:01.626693 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:01.609404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:41169 (local address 127.7.43.62:38017)
0810 19:20:01.609903 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:01.609907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:01.610868 (+ 961us) server_negotiation.cc:407] Connection header received
0810 19:20:01.612736 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:01.612742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:01.612824 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:01.612919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:01.614536 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.615425 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.617116 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.617754 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.617884 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:01.621287 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:01.621292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:01.621295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:01.621328 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:01.623771 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:01.624445 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:01.624451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:01.624454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:01.624579 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:01.624984 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:01.624989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:01.624991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:01.625249 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:01.625382 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:01.626107 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 19:20:01.626327 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
I20260810 19:20:01.627650 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:01.609984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:41169)
0810 19:20:01.610712 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:01.610750 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:01.612471 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:01.613171 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:01.613180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:01.613744 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:01.614339 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:01.614363 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.615578 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.615583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:01.616991 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:01.617004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:01.618090 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:01.618098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:01.618317 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:01.618942 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:01.618970 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:01.621085 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:01.623899 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:01.623908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:01.623925 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:01.624330 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:01.624680 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:01.624686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:01.624690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:01.624875 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:01.625373 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:01.625381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:01.625757 (+ 376us) client_negotiation.cc:769] Sending connection context
0810 19:20:01.626003 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 19:20:01.626522 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":272,"threads_started":1}
I20260810 19:20:01.629076 18291 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38017
I20260810 19:20:01.629362 18291 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:01.630105 18291 heartbeater.cc:511] Master 127.7.43.62:38017 requested a full tablet report, sending...
I20260810 19:20:01.631467 17543 ts_manager.cc:194] Registered new tserver with Master: bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199)
I20260810 19:20:01.632218 17543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:41169
I20260810 19:20:01.642508 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.43.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.61.keytab.
Entry for principal oryx/127.7.43.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.61.keytab.
I20260810 19:20:01.701064 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:41565
--webserver_interface=127.7.43.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.61.keytab
--principal=oryx/127.7.43.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:38017,127.7.43.61:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:02.035883 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:02.036219 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:02.036347 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:02.043730 18304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:02.043853 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:02.043918 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:02.044003 18304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:02.044063 18304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:02.054337 18304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:38017,127.7.43.61:41565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.61.keytab
--principal=oryx/127.7.43.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.43.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
W20260810 19:20:02.054713 18304 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.43.62:38017,127.7.43.61:41565'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 19:20:02.055774 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:02.057128 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:02.069868 18312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:02.069962 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:02.070091 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:02.070811 18304 server_base.cc:1034] running on GCE node
I20260810 19:20:02.071682 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:02.073143 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:02.074440 18304 hybrid_clock.cc:648] HybridClock initialized: now 1786389602074328 us; error 52 us; skew 500 ppm
I20260810 19:20:02.074800 18304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:02.077970 18304 init.cc:377] Logged in from keytab as oryx/127.7.43.61@KRBTEST.COM (short username oryx)
I20260810 19:20:02.079483 18304 webserver.cc:460] Webserver started at http://127.7.43.61:45283/ using document root <none> and password file <none>
I20260810 19:20:02.080204 18304 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:02.080322 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:02.080631 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:02.083817 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/instance:
uuid: "f77cf2ae35eb4c8b986a5f5979d78074"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "4fbc05510790ff4ee8afc4adceade57b"
server_key_iv: "bcac78b22c206ebf95a8762a85227d6c"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.084653 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal/instance:
uuid: "f77cf2ae35eb4c8b986a5f5979d78074"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "4fbc05510790ff4ee8afc4adceade57b"
server_key_iv: "bcac78b22c206ebf95a8762a85227d6c"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.090370 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:20:02.093735 18067 heartbeater.cc:503] Master 127.7.43.62:38017 was elected leader, sending a full tablet report...
I20260810 19:20:02.094341 18332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:02.096189 18304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:02.096370 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
uuid: "f77cf2ae35eb4c8b986a5f5979d78074"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "4fbc05510790ff4ee8afc4adceade57b"
server_key_iv: "bcac78b22c206ebf95a8762a85227d6c"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.096570 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:02.127733 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:02.132040 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:02.132275 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:02.150035 18304 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.61:41565
I20260810 19:20:02.150055 18409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.61:41565 every 8 connection(s)
I20260810 19:20:02.151441 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
I20260810 19:20:02.155224 18413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:02.156418 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 18304
I20260810 19:20:02.164860 18413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.61@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:02.181120 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.158720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.7.43.61:41565)
0810 19:20:02.159545 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.159561 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.159611 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:20:02.161477 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.161508 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.162071 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.162557 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.163911 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.166061 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.167325 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.168135 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.168343 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.173537 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.173548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.173562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.173598 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.176609 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.177413 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.177422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.177439 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.177647 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.178044 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.178050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.178053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.178660 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.178921 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.179385 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.179769 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":338,"threads_started":1}
I20260810 19:20:02.183235 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.167801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36552)
0810 19:20:02.168604 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.168650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.168808 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.169610 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.169616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.170309 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:02.170588 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.170597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.171927 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.171933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.173557 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.173573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.174445 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.174451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.174637 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.175196 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:02.175212 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:02.177537 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:02.180707 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.180718 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.180725 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.181289 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.181700 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.181704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.181708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.181849 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.182426 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:02.182433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:02.182570 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.182729 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.182959 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":343,"threads_started":1}
I20260810 19:20:02.183441 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.167862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.7.43.62:38017)
0810 19:20:02.169216 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.169220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.169235 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:20:02.169283 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.169287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.169382 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.169504 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.170746 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.171788 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.173685 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.174267 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.174385 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.177715 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.177719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.177722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.177760 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.180558 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.181426 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.181431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.181434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.181596 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.181959 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.181965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.181968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.182309 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.182462 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.182884 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.183110 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":1131,"threads_started":1}
I20260810 19:20:02.185173 18413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.61@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
I20260810 19:20:02.196398 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.185831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:53990)
0810 19:20:02.185984 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.185997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.186073 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.186331 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.186334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.186669 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:02.186795 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.186802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.187754 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.187758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.188612 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.188623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.189262 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.189267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.189436 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.190046 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:02.190062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:02.191990 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:02.194470 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.194474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.194476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.194814 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.195094 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.195097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.195099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.195213 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.195657 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:02.195660 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:02.195778 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.195880 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.196082 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 19:20:02.196498 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.186021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.7.43.61:41565)
0810 19:20:02.186148 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.186151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.186164 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:20:02.186199 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.186202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.186251 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.186321 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.186909 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.187640 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.188719 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.189109 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.189199 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.192139 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.192143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.192146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.192173 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.194361 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.194910 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.194914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.194916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.195020 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.195293 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.195296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.195298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.195568 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.195667 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.195958 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.196192 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":43}
I20260810 19:20:02.201274 18413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Bootstrap starting.
I20260810 19:20:02.204324 18413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:02.205332 18413 log.cc:824] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:02.207916 18413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: No bootstrap required, opened a new log
I20260810 19:20:02.215010 18413 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } }
I20260810 19:20:02.215351 18413 raft_consensus.cc:374] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:02.215463 18413 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77cf2ae35eb4c8b986a5f5979d78074, State: Initialized, Role: FOLLOWER
I20260810 19:20:02.216030 18413 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } }
I20260810 19:20:02.216971 18422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } } }
I20260810 19:20:02.217339 18422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:20:02.217888 18413 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:20:02.226842 18446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:20:02.226998 18446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:20:02.230747 18304 master_runner.cc:428] Detected that this master f77cf2ae35eb4c8b986a5f5979d78074 is joining an existing cluster
I20260810 19:20:02.230861 18304 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.43.61:41565
I20260810 19:20:02.251597 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.235053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.7.43.62:38017)
0810 19:20:02.235205 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.235210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.235910 (+ 700us) server_negotiation.cc:407] Connection header received
0810 19:20:02.236189 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.236194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.236318 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.236395 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.237971 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.238986 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.240784 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.241330 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.241424 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.244059 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.244065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.244069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.244110 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.248462 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.249248 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.249254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.249257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.249418 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.249908 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.249913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.249916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.250278 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.250391 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.251006 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.251247 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:20:02.252964 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.234984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36564)
0810 19:20:02.235794 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.235814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.236018 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.236638 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.236643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.237327 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:02.237801 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.237814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.239171 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.239175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.240627 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.240642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.241787 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.241793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.241977 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.242681 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:02.242707 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:02.243841 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:02.248623 (+ 4782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.248629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.248632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.249113 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.249598 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.249602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.249606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.249772 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.250490 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:02.250494 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:02.250715 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.252136 (+ 1421us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.252475 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":508,"threads_started":1}
I20260810 19:20:02.258129 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.234839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.7.43.61:41565)
0810 19:20:02.235046 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.235052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.243002 (+ 7950us) server_negotiation.cc:407] Connection header received
0810 19:20:02.243494 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.243500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.243571 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.243636 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.244979 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.245969 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.247869 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.248572 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.248669 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.251146 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.251152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.251157 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.251195 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.254990 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.255817 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.255822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.255826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.255985 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.256416 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.256422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.256425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.256816 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.256957 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.257435 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.257696 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:20:02.259389 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.235975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:54002)
0810 19:20:02.242892 (+ 6917us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.242918 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.243351 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.243851 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.243856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.244521 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:02.244846 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.244861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.246170 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.246176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.247692 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.247708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.248995 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.249002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.249188 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.249858 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:02.249882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:02.250963 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:02.255143 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.255148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.255152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.255686 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.256131 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.256136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.256139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.256299 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.257041 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:02.257045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:02.257190 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.258586 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.258901 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6735,"spinlock_wait_cycles":30464,"thread_start_us":7032,"threads_started":1}
WARNING: no policy specified for oryx/127.7.43.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.60@KRBTEST.COM" created.
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.62@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
I20260810 19:20:02.287339 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.259908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:54004)
0810 19:20:02.269599 (+ 9691us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.269657 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.269925 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.270399 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.270408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.271566 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:02.271889 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.271904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.273263 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.273269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.274998 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.275016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.275939 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.275945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.276169 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.276921 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:02.276948 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:02.280141 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:02.284210 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.284223 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.284232 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.284765 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.285211 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.285216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.285220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.285391 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.286148 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:02.286157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:02.286349 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.286567 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.286935 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9376,"thread_start_us":9204,"threads_started":1}
I20260810 19:20:02.287461 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.260364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.7.43.61:41565)
0810 19:20:02.260553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.260558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.269771 (+ 9213us) server_negotiation.cc:407] Connection header received
0810 19:20:02.270159 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.270163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.270235 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.270323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.272092 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.273103 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.275155 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.275763 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.275882 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.280349 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.280357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.280361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.280402 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.284050 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.284901 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.284908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.284911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.285079 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.285549 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.285554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.285558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.285989 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.286133 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.286696 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.286947 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:20:02.290031 17543 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.43.61:41565: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074"
member_type: NON_VOTER
last_known_addr {
host: "127.7.43.61"
port: 41565
}
attrs {
promote: true
}
}
dest_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf"
cas_config_opid_index: -1
I20260810 19:20:02.291769 17543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } }
I20260810 19:20:02.299454 18473 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f77cf2ae35eb4c8b986a5f5979d78074 (127.7.43.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } }
I20260810 19:20:02.300660 18370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:02.304345 18473 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.43.61:41565
I20260810 19:20:02.304898 18473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } }
I20260810 19:20:02.305254 18370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Refusing update from remote peer 83bb9449eafe4dcd909a5eaf6160fbbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 19:20:02.305284 18473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:02.305377 18304 master.cc:537] Master@127.7.43.61:41565 shutting down...
I20260810 19:20:02.307250 18473 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:20:02.310273 17523 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.60.keytab.
Entry for principal oryx/127.7.43.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.60.keytab.
I20260810 19:20:02.312173 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:36351
--webserver_interface=127.7.43.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.60.keytab
--principal=oryx/127.7.43.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:02.315402 17523 consensus_peers.cc:487] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf -> Peer f77cf2ae35eb4c8b986a5f5979d78074 (127.7.43.61:41565): Couldn't send request to peer f77cf2ae35eb4c8b986a5f5979d78074. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 19:20:02.319924 18304 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:02.320389 18304 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:02.320591 18304 tablet_replica.cc:330] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: stopping tablet replica
I20260810 19:20:02.327879 18304 master.cc:552] Master@127.7.43.61:41565 shutdown complete.
I20260810 19:20:02.328027 18304 master_runner.cc:305] Clearing existing system tablet
I20260810 19:20:02.328522 18304 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:02.328630 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:02.331701 18304 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:20:02.333487 18485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:02.333863 18304 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:02.334160 18304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 19:20:02.334285 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
uuid: "f77cf2ae35eb4c8b986a5f5979d78074"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "4fbc05510790ff4ee8afc4adceade57b"
server_key_iv: "bcac78b22c206ebf95a8762a85227d6c"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.339764 18304 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:02.346812 18304 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 19:20:02.347014 18304 log.cc:1196] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 19:20:02.347488 18304 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Deleting consensus metadata
I20260810 19:20:02.347775 18304 master_runner.cc:315] Copying system tablet from 127.7.43.62:38017
I20260810 19:20:02.351557 18304 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: tablet copy: Beginning tablet copy session from remote peer at address 127.7.43.62:38017
I20260810 19:20:02.368026 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.352823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.7.43.62:38017)
0810 19:20:02.353028 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.353035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.354062 (+ 1027us) server_negotiation.cc:407] Connection header received
0810 19:20:02.354251 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.354257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.354364 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.354479 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.355531 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.356651 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.358478 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.359116 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.359246 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.361455 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:02.361461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:02.361466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:02.361508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:02.364961 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.365767 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.365773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.365776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.365942 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:02.366343 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:02.366348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:02.366351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:02.366782 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:02.366965 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.367413 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.367648 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:20:02.368054 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.352660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36570)
0810 19:20:02.353916 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.353946 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.354106 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.354521 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.354527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.355074 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:02.355359 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.355372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.356805 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.356811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.358309 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.358324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.359265 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.359271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.359445 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.360149 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:02.360173 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:02.361217 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:02.365092 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.365098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.365102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.365645 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.366058 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:02.366063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:02.366067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:02.366234 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:02.366935 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:02.366941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:02.367121 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.367268 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.367598 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":1021,"threads_started":1}
I20260810 19:20:02.369665 17582 tablet_copy_service.cc:140] P 83bb9449eafe4dcd909a5eaf6160fbbf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f77cf2ae35eb4c8b986a5f5979d78074 ({username='oryx', principal='oryx/127.7.43.61@KRBTEST.COM'} at 127.0.0.1:36570)
I20260810 19:20:02.370172 17582 tablet_copy_service.cc:161] P 83bb9449eafe4dcd909a5eaf6160fbbf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f77cf2ae35eb4c8b986a5f5979d78074 at {username='oryx', principal='oryx/127.7.43.61@KRBTEST.COM'} at 127.0.0.1:36570: session id = f77cf2ae35eb4c8b986a5f5979d78074-00000000000000000000000000000000
I20260810 19:20:02.375473 17582 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 19:20:02.378744 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 19:20:02.385818 18304 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: tablet copy: Starting download of 0 data blocks...
I20260810 19:20:02.386231 18304 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: tablet copy: Starting download of 1 WAL segments...
I20260810 19:20:02.390324 18304 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 19:20:02.395217 17582 tablet_copy_service.cc:342] P 83bb9449eafe4dcd909a5eaf6160fbbf: Request end of tablet copy session f77cf2ae35eb4c8b986a5f5979d78074-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.43.61@KRBTEST.COM'} at 127.0.0.1:36570
I20260810 19:20:02.395517 17582 tablet_copy_service.cc:433] P 83bb9449eafe4dcd909a5eaf6160fbbf: ending tablet copy session f77cf2ae35eb4c8b986a5f5979d78074-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f77cf2ae35eb4c8b986a5f5979d78074
I20260810 19:20:02.408494 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:02.411509 18498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:02.413221 18499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:02.420871 18503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:02.422739 18304 server_base.cc:1034] running on GCE node
I20260810 19:20:02.423193 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:02.423909 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:02.425215 18304 hybrid_clock.cc:648] HybridClock initialized: now 1786389602425142 us; error 38 us; skew 500 ppm
I20260810 19:20:02.425469 18304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:02.427841 18304 init.cc:377] Logged in from keytab as oryx/127.7.43.61@KRBTEST.COM (short username oryx)
I20260810 19:20:02.429222 18304 webserver.cc:460] Webserver started at http://127.7.43.61:44477/ using document root <none> and password file <none>
I20260810 19:20:02.429610 18304 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:02.429697 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:02.432835 18304 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:20:02.434983 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:02.435464 18304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:20:02.435637 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
uuid: "f77cf2ae35eb4c8b986a5f5979d78074"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "4fbc05510790ff4ee8afc4adceade57b"
server_key_iv: "bcac78b22c206ebf95a8762a85227d6c"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.435824 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:02.456218 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:02.460448 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:02.482123 18304 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.61:41565
I20260810 19:20:02.482149 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.61:41565 every 8 connection(s)
I20260810 19:20:02.483209 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
I20260810 19:20:02.488507 18586 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:20:02.492508 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Bootstrap starting.
I20260810 19:20:02.511969 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:20:02.512604 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Bootstrap complete.
I20260810 19:20:02.513756 18586 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } }
I20260810 19:20:02.514050 18586 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f77cf2ae35eb4c8b986a5f5979d78074, State: Initialized, Role: LEARNER
I20260810 19:20:02.514377 18586 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } }
I20260810 19:20:02.515322 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } }
I20260810 19:20:02.515592 18586 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:02.515771 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: LEARNER
I20260810 19:20:02.524669 18609 catalog_manager.cc:1259] Loaded cluster ID: 84754b7befde449a98e32871b6cfdf53
I20260810 19:20:02.524787 18609 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: loading cluster ID for follower catalog manager: success
I20260810 19:20:02.528375 18609 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: acquiring CA information for follower catalog manager: success
I20260810 19:20:02.531282 18609 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:20:02.634799 18291 heartbeater.cc:503] Master 127.7.43.62:38017 was elected leader, sending a full tablet report...
W20260810 19:20:02.848385 18481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:02.848681 18481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:02.848800 18481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:02.858307 18481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:02.858497 18481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:02.858616 18481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:02.858691 18481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:02.858753 18481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:02.871166 18481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.60.keytab
--principal=oryx/127.7.43.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:36351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.43.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:02.872527 18481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:02.873992 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:02.888615 18619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:02.888468 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:02.889566 18623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:02.890154 18481 server_base.cc:1034] running on GCE node
I20260810 19:20:02.891314 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:02.893007 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:02.894218 18481 hybrid_clock.cc:648] HybridClock initialized: now 1786389602894161 us; error 44 us; skew 500 ppm
I20260810 19:20:02.894555 18481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:02 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:02.898205 18481 init.cc:377] Logged in from keytab as oryx/127.7.43.60@KRBTEST.COM (short username oryx)
I20260810 19:20:02.899685 18481 webserver.cc:460] Webserver started at http://127.7.43.60:37595/ using document root <none> and password file <none>
I20260810 19:20:02.900347 18481 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:02.900446 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:02.900709 18481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:02.904327 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/instance:
uuid: "d5edef1593d348e5808a6aa287db21bf"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "29f9cae9f2c8699d3cd36a52c7fbee06"
server_key_iv: "d7e9b95d14d9e1ad404024cc795d1062"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.905284 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal/instance:
uuid: "d5edef1593d348e5808a6aa287db21bf"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "29f9cae9f2c8699d3cd36a52c7fbee06"
server_key_iv: "d7e9b95d14d9e1ad404024cc795d1062"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.911755 18481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 19:20:02.917125 18637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:02.918659 18481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 19:20:02.918875 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
uuid: "d5edef1593d348e5808a6aa287db21bf"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "29f9cae9f2c8699d3cd36a52c7fbee06"
server_key_iv: "d7e9b95d14d9e1ad404024cc795d1062"
server_key_version: "encryptionkey@0"
I20260810 19:20:02.919095 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:02.922221 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.913869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:54012)
0810 19:20:02.914711 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:02.914734 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:02.914886 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:02.915466 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:02.915470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:02.915498 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:02.915753 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.915770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.917407 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.917411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.919500 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:02.919516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.921124 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.921130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:02.921327 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.921439 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 19:20:02.921562 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 19:20:02.921714 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":538,"threads_started":1}
I20260810 19:20:02.922259 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.914044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.7.43.61:41565)
0810 19:20:02.915104 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.915109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.915124 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:20:02.915183 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.915186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.915257 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.915385 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:02.915950 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.917239 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.919685 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.920939 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:02.921080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:02.921319 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:02.921589 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 19:20:02.921760 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":802,"threads_started":1}
I20260810 19:20:02.923637 18548 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 19:20:02.924580 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83bb9449eafe4dcd909a5eaf6160fbbf. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } }
I20260810 19:20:02.924906 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: LEARNER
I20260810 19:20:02.936476 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:02.942451 18481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:02.942875 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:02.980340 18481 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.60:36351
I20260810 19:20:02.980387 18732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.60:36351 every 8 connection(s)
I20260810 19:20:02.982239 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
I20260810 19:20:02.986727 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:02.986819 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 18481
I20260810 19:20:02.999357 18733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.60@KRBTEST.COM
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.60@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:03.017200 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:02.989297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.7.43.60:36351)
0810 19:20:02.990498 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:02.990510 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:02.990563 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:20:02.993154 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:02.993177 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:02.993641 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:02.994038 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:02.996347 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:02.998500 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.000238 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.001267 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.001433 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.007492 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.007500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.007511 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.007551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.011629 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.012654 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.012663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.012670 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.012942 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.013473 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.013477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.013479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.014269 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.014659 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.015113 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.015594 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":658,"threads_started":1}
I20260810 19:20:03.024240 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.002212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36584)
0810 19:20:03.003297 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.003353 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.003512 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.004354 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.004362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.005466 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.006276 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.006300 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.007706 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.007710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.009644 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.009658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.010857 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.010876 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.011065 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.011824 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.011863 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.015088 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.020946 (+ 5858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.020956 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.020962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.021700 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.022147 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.022151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.022155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.022418 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.023002 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.023011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.023237 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.023435 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.023685 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":655,"threads_started":1}
I20260810 19:20:03.024309 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.002344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.7.43.62:38017)
0810 19:20:03.003938 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.003943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.003961 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:20:03.004014 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.004025 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.004105 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.004212 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.006503 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.007535 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.009814 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.010528 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.010659 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.015348 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.015356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.015362 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.015405 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.020791 (+ 5386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.021825 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.021831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.021835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.022037 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.022506 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.022509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.022511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.022860 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.022976 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.023594 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.023766 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":1365,"threads_started":1}
I20260810 19:20:03.025723 18733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.60@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
I20260810 19:20:03.041589 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.026609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:54024)
0810 19:20:03.026786 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.026801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.026887 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.027140 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.027143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.027625 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.027846 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.027855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.029142 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.029145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.030277 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.030298 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.031197 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.031201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.031338 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.031897 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.031914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.034779 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.039002 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.039006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.039010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.039579 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.040016 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.040019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.040021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.040198 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.040698 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.040702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.040882 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.040992 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.041262 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:20:03.041786 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.026503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.7.43.61:41565)
0810 19:20:03.026713 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.026720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.026866 (+ 146us) server_negotiation.cc:407] Connection header received
0810 19:20:03.026973 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.026977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.027042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.027116 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.027969 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.029002 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.030420 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.031073 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.031172 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.034952 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.034958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.034961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.034995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.038837 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.039682 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.039686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.039689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.039923 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.040280 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.040283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.040285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.040615 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.040724 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.041070 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.041297 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:20:03.042958 18733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389602, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.60@KRBTEST.COM for oryx/127.7.43.60@KRBTEST.COM
I20260810 19:20:03.058266 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.043713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.7.43.60:36351)
0810 19:20:03.043843 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.043847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.043992 (+ 145us) server_negotiation.cc:407] Connection header received
0810 19:20:03.044076 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.044079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.044131 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.044199 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.045071 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.045794 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.047189 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.047724 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.047837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.051772 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.051777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.051780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.051807 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.055266 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.056029 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.056034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.056036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.056170 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.056640 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.056644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.056646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.057030 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.057158 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.057508 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.057912 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":45}
I20260810 19:20:03.058329 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.043638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.0.0.1:60858)
0810 19:20:03.043899 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.043914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.044002 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.044225 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.044228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.044718 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.044958 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.044967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.045897 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.045901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.047069 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.047081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.047913 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.047918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.048175 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.048809 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.048826 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.051583 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.055386 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.055391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.055395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.055921 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.056272 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.056276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.056280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.056537 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.057140 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.057145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.057298 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.057416 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.057705 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 19:20:03.064517 18733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Bootstrap starting.
I20260810 19:20:03.069234 18733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:03.071069 18733 log.cc:824] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:03.074950 18733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: No bootstrap required, opened a new log
I20260810 19:20:03.086262 18733 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } }
I20260810 19:20:03.086719 18733 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:03.086935 18733 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5edef1593d348e5808a6aa287db21bf, State: Initialized, Role: FOLLOWER
I20260810 19:20:03.087576 18733 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } }
I20260810 19:20:03.089005 18747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } } }
I20260810 19:20:03.089653 18747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:20:03.089871 18733 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:20:03.105152 18767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:20:03.105372 18767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:20:03.109398 18481 master_runner.cc:428] Detected that this master d5edef1593d348e5808a6aa287db21bf is joining an existing cluster
I20260810 19:20:03.109719 18481 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.43.60:36351
I20260810 19:20:03.124951 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 17634
I20260810 19:20:03.148521 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.116772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.7.43.61:41565)
0810 19:20:03.117007 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.117013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.119111 (+ 2098us) server_negotiation.cc:407] Connection header received
0810 19:20:03.120412 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.120419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.120523 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.120597 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.128117 (+ 7520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.129639 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.133071 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.134245 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.134356 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.138809 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.138818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.138823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.138870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.143788 (+ 4918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.145185 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.145191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.145194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.145501 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.146112 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.146117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.146120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.146675 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.146809 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.147661 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.147981 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:20:03.151234 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.116601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.7.43.62:38017)
0810 19:20:03.116811 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.116818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.120599 (+ 3781us) server_negotiation.cc:407] Connection header received
0810 19:20:03.120903 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.120910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.121042 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.121124 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.127861 (+ 6737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.129277 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.132658 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.133420 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.133563 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.139258 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.139265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.139270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.139315 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.144278 (+ 4963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.145621 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.145626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.145630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.145926 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.146513 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.146519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.146523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.148668 (+ 2145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.148867 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.150591 (+ 1724us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.150876 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:20:03.153051 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.117067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:54026)
0810 19:20:03.118976 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.118997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.120186 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.120873 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.120878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.122619 (+ 1741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.127898 (+ 5279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.127921 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.129938 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.129945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.132873 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.132892 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.134838 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.134846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.135117 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.136181 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.136207 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.138477 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.143987 (+ 5510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.143992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.143997 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.145042 (+ 1045us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.145657 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.145662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.145665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.145976 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.146911 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.146916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.147326 (+ 410us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.149078 (+ 1752us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.149471 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":1328,"threads_started":1}
I20260810 19:20:03.153494 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.117871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36588)
0810 19:20:03.120483 (+ 2612us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.120506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.120736 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.121343 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.121348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.122804 (+ 1456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.127645 (+ 4841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.127661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.129538 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.129544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.132449 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.132471 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.133974 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.133982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.134263 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.135184 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.135210 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.138955 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.144458 (+ 5503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.144463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.144467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.145449 (+ 982us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.146074 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.146078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.146081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.146372 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.148951 (+ 2579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.148955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.149337 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.150395 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.152485 (+ 2090us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2418,"mutex_wait_us":3883,"thread_start_us":7641,"threads_started":1}
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.1@KRBTEST.COM
I20260810 19:20:03.170751 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.119212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.7.43.60:36351)
0810 19:20:03.119485 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.119491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.126135 (+ 6644us) server_negotiation.cc:407] Connection header received
0810 19:20:03.126414 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.126419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.126609 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.126689 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.146709 (+ 20020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.148157 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.153655 (+ 5498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.154899 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.155006 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.159159 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.159167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.159172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.159212 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.164338 (+ 5126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.165286 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.165292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.165296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.165579 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.166097 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.166103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.166107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.166739 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.166926 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.169089 (+ 2163us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.169306 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:20:03.171397 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.118993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.0.0.1:60866)
0810 19:20:03.126043 (+ 7050us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.126063 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.126253 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.129909 (+ 3656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.129916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.144682 (+ 14766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.146563 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.146580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.149687 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.149693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.152169 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.152185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.155444 (+ 3259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.155452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.155706 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.156633 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.156664 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.158932 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.164498 (+ 5566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.164503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.164507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.165167 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.165712 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.165718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.165722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.165925 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.168617 (+ 2692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.168622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.168808 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.169739 (+ 931us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.170083 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6832,"mutex_wait_us":5,"thread_start_us":1013,"threads_started":1}
I20260810 19:20:03.175911 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.126554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.7.43.1:36603)
0810 19:20:03.142053 (+ 15499us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.142065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.142102 (+ 37us) server_negotiation.cc:407] Connection header received
0810 19:20:03.142178 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.142188 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.142496 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.145619 (+ 3123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.146129 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.148471 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.150713 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.151904 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.152023 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.167276 (+ 15253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.167298 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.167319 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.167373 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.171921 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.172915 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.172926 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.172929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.173100 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.173757 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.173761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.173763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.174168 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.174605 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.174991 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.175304 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15231,"thread_start_us":16096,"threads_started":1}
I20260810 19:20:03.177850 17842 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.62@KRBTEST.COM for oryx/127.7.43.60@KRBTEST.COM
I20260810 19:20:03.189744 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.160608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.0.0.1:60876)
0810 19:20:03.160889 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.160912 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.161022 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.168174 (+ 7152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.168181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.168799 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.169162 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.169177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.172034 (+ 2857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.172040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.173552 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.173570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.175901 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.175908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.176189 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.177036 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.177059 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.179750 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.186067 (+ 6317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.186074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.186078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.186895 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.187469 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.187474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.187478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.187734 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.188473 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.188479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.188703 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.188865 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.189189 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 19:20:03.190109 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.160703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.7.43.60:36351)
0810 19:20:03.167727 (+ 7024us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.167733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.167758 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:20:03.167864 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.167869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.167967 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.168095 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.169410 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.171647 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.174685 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.175660 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.176614 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.180104 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.180113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.180118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.180158 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.185868 (+ 5710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.187087 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.187093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.187097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.187320 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.187905 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.187909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.187913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.188319 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.189201 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.189340 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.189645 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6838,"thread_start_us":6546,"threads_started":1}
I20260810 19:20:03.191885 17543 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.43.60:36351: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d5edef1593d348e5808a6aa287db21bf"
member_type: NON_VOTER
last_known_addr {
host: "127.7.43.60"
port: 36351
}
attrs {
promote: true
}
}
dest_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf"
cas_config_opid_index: 5
I20260810 19:20:03.192951 17543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } }
I20260810 19:20:03.195672 18548 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 LEARNER]: Refusing update from remote peer 83bb9449eafe4dcd909a5eaf6160fbbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260810 19:20:03.196029 18653 tablet.cc:2369] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:03.196687 18630 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 19:20:03.199797 18630 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d5edef1593d348e5808a6aa287db21bf (127.7.43.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.205669 18695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:03.210047 18630 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.43.60:36351
I20260810 19:20:03.210836 18481 master.cc:537] Master@127.7.43.60:36351 shutting down...
I20260810 19:20:03.210460 18785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.211019 18785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:03.214406 18695 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Refusing update from remote peer 83bb9449eafe4dcd909a5eaf6160fbbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 19:20:03.214888 18548 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d5edef1593d348e5808a6aa287db21bf (127.7.43.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.219632 18785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:03.224535 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: NON_VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: true } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.224872 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: LEARNER
I20260810 19:20:03.226754 18630 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: attempting to promote NON_VOTER f77cf2ae35eb4c8b986a5f5979d78074 to VOTER
I20260810 19:20:03.227630 18630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } }
I20260810 19:20:03.231315 18547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 LEARNER]: Refusing update from remote peer 83bb9449eafe4dcd909a5eaf6160fbbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260810 19:20:03.231880 17523 consensus_peers.cc:487] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf -> Peer d5edef1593d348e5808a6aa287db21bf (127.7.43.60:36351): Couldn't send request to peer d5edef1593d348e5808a6aa287db21bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 19:20:03.232239 18791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 19:20:03.236106 18547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f77cf2ae35eb4c8b986a5f5979d78074 (127.7.43.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.236825 18789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f77cf2ae35eb4c8b986a5f5979d78074 (127.7.43.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.240876 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.240927 18789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.241206 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:20:03.241236 18789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:03.241775 18785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.242133 18785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:03.255635 18481 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:03.256063 18481 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:03.256279 18481 tablet_replica.cc:330] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: stopping tablet replica
I20260810 19:20:03.262780 18481 master.cc:552] Master@127.7.43.60:36351 shutdown complete.
I20260810 19:20:03.262872 18481 master_runner.cc:305] Clearing existing system tablet
I20260810 19:20:03.263242 18481 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:03.263345 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:03.265795 18481 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 19:20:03.267880 18794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:03.268254 18481 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:03.268493 18481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260810 19:20:03.268630 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
uuid: "d5edef1593d348e5808a6aa287db21bf"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "29f9cae9f2c8699d3cd36a52c7fbee06"
server_key_iv: "d7e9b95d14d9e1ad404024cc795d1062"
server_key_version: "encryptionkey@0"
I20260810 19:20:03.272305 18481 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:03.277350 18481 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 19:20:03.277594 18481 log.cc:1196] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 19:20:03.278187 18481 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Deleting consensus metadata
I20260810 19:20:03.278427 18481 master_runner.cc:315] Copying system tablet from 127.7.43.62:38017
I20260810 19:20:03.281030 18481 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: tablet copy: Beginning tablet copy session from remote peer at address 127.7.43.62:38017
I20260810 19:20:03.300305 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.283104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.7.43.62:38017)
0810 19:20:03.283448 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.283454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.283869 (+ 415us) server_negotiation.cc:407] Connection header received
0810 19:20:03.284140 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.284145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.284212 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.284299 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.285575 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.286569 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.288265 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.289009 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.289323 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.294011 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:03.294018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:03.294021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:03.294055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:03.297084 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.297851 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.297857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.297861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.298102 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:03.298534 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:03.298538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:03.298540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:03.298903 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:03.299081 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.299550 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.299765 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260810 19:20:03.300485 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.282666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36596)
0810 19:20:03.283731 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.283770 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.283948 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.284349 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.284353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.284990 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.285394 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.285405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.286700 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.286706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.288129 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.288142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.289199 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.289206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.289453 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.290379 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:03.290404 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:03.293772 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:03.297215 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.297220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.297224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.297737 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.298217 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:03.298221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:03.298223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:03.298419 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:03.299050 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:03.299055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:03.299244 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.299368 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.299747 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":799,"threads_started":1}
I20260810 19:20:03.301371 17582 tablet_copy_service.cc:140] P 83bb9449eafe4dcd909a5eaf6160fbbf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d5edef1593d348e5808a6aa287db21bf ({username='oryx', principal='oryx/127.7.43.60@KRBTEST.COM'} at 127.0.0.1:36596)
I20260810 19:20:03.301646 17582 tablet_copy_service.cc:161] P 83bb9449eafe4dcd909a5eaf6160fbbf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d5edef1593d348e5808a6aa287db21bf at {username='oryx', principal='oryx/127.7.43.60@KRBTEST.COM'} at 127.0.0.1:36596: session id = d5edef1593d348e5808a6aa287db21bf-00000000000000000000000000000000
I20260810 19:20:03.305161 17582 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 19:20:03.307053 18481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 19:20:03.313738 18481 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: tablet copy: Starting download of 0 data blocks...
I20260810 19:20:03.314110 18481 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: tablet copy: Starting download of 1 WAL segments...
I20260810 19:20:03.317130 18481 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 19:20:03.322256 17582 tablet_copy_service.cc:342] P 83bb9449eafe4dcd909a5eaf6160fbbf: Request end of tablet copy session d5edef1593d348e5808a6aa287db21bf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.43.60@KRBTEST.COM'} at 127.0.0.1:36596
I20260810 19:20:03.322407 17582 tablet_copy_service.cc:433] P 83bb9449eafe4dcd909a5eaf6160fbbf: ending tablet copy session d5edef1593d348e5808a6aa287db21bf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d5edef1593d348e5808a6aa287db21bf
I20260810 19:20:03.327339 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 17634
I20260810 19:20:03.343165 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:03.346249 18810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:03.346692 18812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:03.361490 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:03.362084 18481 server_base.cc:1034] running on GCE node
I20260810 19:20:03.362428 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:03.362859 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:36603
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:38017
--webserver_port=46775
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:20:03.365831 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:03.367172 18481 hybrid_clock.cc:648] HybridClock initialized: now 1786389603367123 us; error 42 us; skew 500 ppm
I20260810 19:20:03.367446 18481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389603, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:03.369697 18481 init.cc:377] Logged in from keytab as oryx/127.7.43.60@KRBTEST.COM (short username oryx)
I20260810 19:20:03.373601 18481 webserver.cc:460] Webserver started at http://127.7.43.60:35429/ using document root <none> and password file <none>
I20260810 19:20:03.373937 18481 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:03.374048 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:03.377331 18481 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 19:20:03.379359 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:03.379803 18481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 19:20:03.379979 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
uuid: "d5edef1593d348e5808a6aa287db21bf"
format_stamp: "Formatted at 2026-08-10 19:20:02 on dist-test-slave-7lbf"
server_key: "29f9cae9f2c8699d3cd36a52c7fbee06"
server_key_iv: "d7e9b95d14d9e1ad404024cc795d1062"
server_key_version: "encryptionkey@0"
I20260810 19:20:03.380259 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:03.401858 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:03.406052 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:03.432700 18481 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.60:36351
I20260810 19:20:03.432780 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.60:36351 every 8 connection(s)
I20260810 19:20:03.433802 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
I20260810 19:20:03.439687 18904 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:20:03.443531 18904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Bootstrap starting.
I20260810 19:20:03.462594 18904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:20:03.464247 18904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: Bootstrap complete.
I20260810 19:20:03.465490 18904 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } }
I20260810 19:20:03.465847 18904 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d5edef1593d348e5808a6aa287db21bf, State: Initialized, Role: LEARNER
I20260810 19:20:03.466159 18904 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } }
I20260810 19:20:03.467393 18904 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:03.468917 18908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.469731 18908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: This master's current role is: LEARNER
I20260810 19:20:03.499096 18929 catalog_manager.cc:1259] Loaded cluster ID: 84754b7befde449a98e32871b6cfdf53
I20260810 19:20:03.499336 18929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: loading cluster ID for follower catalog manager: success
I20260810 19:20:03.503000 18929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: acquiring CA information for follower catalog manager: success
I20260810 19:20:03.505956 18929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:20:03.793458 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.785835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.0.0.1:60882)
0810 19:20:03.786699 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.786717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.786860 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.787271 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.787274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.787296 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:03.787508 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.787516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.789352 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.789356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.790818 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.790836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.792853 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.792857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:03.792986 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.793013 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:20:03.793075 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 19:20:03.793172 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":649,"threads_started":1}
I20260810 19:20:03.793861 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.786007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.7.43.60:36351)
0810 19:20:03.786586 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.786593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.786809 (+ 216us) server_negotiation.cc:407] Connection header received
0810 19:20:03.786988 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.786994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.787103 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.787260 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:03.787642 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.789155 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.790994 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:03.792671 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:03.792817 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:03.793061 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:03.793114 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 19:20:03.793329 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":252,"threads_started":1}
I20260810 19:20:03.794795 18863 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 19:20:03.795449 18908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83bb9449eafe4dcd909a5eaf6160fbbf. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:03.795729 18908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: This master's current role is: LEARNER
W20260810 19:20:03.811436 18821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:03.811725 18821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:03.811935 18821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:03.821321 18821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:03.821430 18821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:03.821606 18821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:03.833202 18821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:36603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=46775
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:03.834818 18821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:03.836535 18821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:03.855114 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:03.855103 18947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:03.855114 18953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:03.855857 18821 server_base.cc:1034] running on GCE node
I20260810 19:20:03.856534 18821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:03.857563 18821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:03.858846 18821 hybrid_clock.cc:648] HybridClock initialized: now 1786389603858734 us; error 44 us; skew 500 ppm
I20260810 19:20:03.859247 18821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:03 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389603, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:03.863746 18821 init.cc:377] Logged in from keytab as oryx/127.7.43.1@KRBTEST.COM (short username oryx)
I20260810 19:20:03.865685 18821 webserver.cc:460] Webserver started at http://127.7.43.1:46775/ using document root <none> and password file <none>
I20260810 19:20:03.866432 18821 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:03.866573 18821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:03.873364 18821 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:20:03.877319 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:03.878921 18821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:20:03.879130 18821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "c64aaeddbe424cea978f4db16dfd3778"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "8f410f0145661b99cd4391696ffcbe25"
server_key_iv: "bef443d5225d4340150e48af963384fd"
server_key_version: "encryptionkey@0"
I20260810 19:20:03.880555 18821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:03.895754 18821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:03.901563 18821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:03.901985 18821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:03.903625 18821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:03.905825 18821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:03.905997 18821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:03.906148 18821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:03.906316 18821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:03.982429 18821 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:36603
I20260810 19:20:03.982542 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:36603 every 8 connection(s)
I20260810 19:20:03.984390 18821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:03.994161 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 18821
I20260810 19:20:03.994616 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 17862
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389603, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.60@KRBTEST.COM
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389603, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:04.023336 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.988778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:33911 (local address 127.7.43.60:36351)
0810 19:20:03.988947 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.988952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:03.991942 (+ 2990us) server_negotiation.cc:407] Connection header received
0810 19:20:03.996068 (+ 4126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:03.996076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:03.996177 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:03.996251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:03.999606 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.001063 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.006102 (+ 5039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.006782 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.006884 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.013504 (+ 6620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.013511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.013534 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.013580 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.018580 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.019661 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.019667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.019671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.019936 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.020620 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.020627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.020630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.021047 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.021211 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.022403 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.022697 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:20:04.026566 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.990097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.7.43.1:33911)
0810 19:20:03.991752 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:03.991819 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:03.995693 (+ 3874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:03.996979 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:03.996992 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:03.998161 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:03.999361 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:03.999391 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.001381 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.001387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.004060 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.004080 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.007285 (+ 3205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.007293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.007711 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.008671 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:04.008713 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:04.013174 (+ 4461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:04.018743 (+ 5569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.018752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.018770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.019511 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.020142 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.020147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.020150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.020474 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.021313 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:04.021322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:04.021876 (+ 554us) client_negotiation.cc:769] Sending connection context
0810 19:20:04.023917 (+ 2041us) client_negotiation.cc:240] Negotiation successful
0810 19:20:04.024600 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":52,"thread_start_us":801,"threads_started":1}
I20260810 19:20:04.027554 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.990097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:33145)
0810 19:20:04.005259 (+ 15162us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:04.005283 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:04.005436 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:04.006021 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:04.006027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:04.007272 (+ 1245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:04.008336 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.008353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.010374 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.010380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.012492 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.012511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.014279 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.014287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.014539 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.015224 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:04.015242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:04.018579 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:04.023609 (+ 5030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.023614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.023617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.024353 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.024896 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.024900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.024904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.025150 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.026066 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:04.026072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:04.026257 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 19:20:04.026423 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 19:20:04.026793 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14980,"mutex_wait_us":74,"spinlock_wait_cycles":16256,"thread_start_us":27894,"threads_started":1}
I20260810 19:20:04.028896 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.988430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:33145 (local address 127.7.43.62:38017)
0810 19:20:03.989176 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:03.989182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.005413 (+ 16231us) server_negotiation.cc:407] Connection header received
0810 19:20:04.005572 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.005578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.005687 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.005806 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.008566 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.010165 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.012689 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.013708 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.013824 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.018812 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.018819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.018824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.018880 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.023443 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.024529 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.024535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.024538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.024744 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.025304 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.025309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.025312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.025788 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.025968 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.028043 (+ 2075us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.028293 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":473,"threads_started":1}
I20260810 19:20:04.029273 19173 heartbeater.cc:347] Connected to a master server at 127.7.43.60:36351
I20260810 19:20:04.029757 19173 heartbeater.cc:464] Registering TS with master...
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.2@KRBTEST.COM
I20260810 19:20:04.030326 19176 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38017
I20260810 19:20:04.030567 19176 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:04.031405 19173 heartbeater.cc:511] Master 127.7.43.60:36351 requested a full tablet report, sending...
I20260810 19:20:04.031601 19176 heartbeater.cc:511] Master 127.7.43.62:38017 requested a full tablet report, sending...
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389603, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
I20260810 19:20:04.033758 17543 ts_manager.cc:194] Re-registered known tserver with Master: c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603)
I20260810 19:20:04.034610 18850 ts_manager.cc:194] Registered new tserver with Master: c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603)
I20260810 19:20:04.035140 17543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:33145
I20260810 19:20:04.041221 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.992127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:43037 (local address 127.7.43.61:41565)
0810 19:20:04.005958 (+ 13831us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:04.005977 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.018162 (+ 12185us) server_negotiation.cc:407] Connection header received
0810 19:20:04.018755 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.018760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.018905 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.018992 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.021154 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.022544 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.025107 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.026047 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.026145 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.034061 (+ 7916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.034067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.034072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.034117 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.038347 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.039066 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.039072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.039075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.039192 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.039612 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.039618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.039621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.039982 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.040115 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.040490 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.040700 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13698,"thread_start_us":13590,"threads_started":1}
I20260810 19:20:04.041220 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.992025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.7.43.1:43037)
0810 19:20:04.018054 (+ 26029us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:04.018076 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:04.018596 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:04.019341 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:04.019346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:04.020261 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:04.020970 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.020985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.022775 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.022780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.024937 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.024953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.026571 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.026580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.026884 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.027632 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:04.027802 (+ 170us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:04.033797 (+ 5995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:04.038519 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.038524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.038527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.038946 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.039304 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.039310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.039314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.039500 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.040131 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:04.040136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:04.040253 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 19:20:04.040369 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 19:20:04.040701 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25823,"thread_start_us":12699,"threads_started":1}
I20260810 19:20:04.042145 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:03.995985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.7.43.2:35867)
0810 19:20:04.014719 (+ 18734us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:04.014729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.014762 (+ 33us) server_negotiation.cc:407] Connection header received
0810 19:20:04.014831 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.014840 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.015189 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.015359 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.018002 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.020364 (+ 2362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.022682 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.024033 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.030290 (+ 6257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.031995 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.032008 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.032021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.032055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.038464 (+ 6409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.039321 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.039329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.039331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.039506 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.040203 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.040209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.040211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.040650 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.041026 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.041349 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.041663 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18479,"thread_start_us":18143,"threads_started":1}
I20260810 19:20:04.042549 19178 heartbeater.cc:347] Connected to a master server at 127.7.43.61:41565
I20260810 19:20:04.042701 19178 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:04.043285 19178 heartbeater.cc:511] Master 127.7.43.61:41565 requested a full tablet report, sending...
I20260810 19:20:04.043610 18060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:04.045285 18537 ts_manager.cc:194] Registered new tserver with Master: c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603)
I20260810 19:20:04.128654 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 17862
I20260810 19:20:04.153452 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:35867
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:38017
--webserver_port=40483
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:20:04.250411 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: NON_VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: true } } }
I20260810 19:20:04.250882 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:04.252403 19196 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: attempting to promote NON_VOTER d5edef1593d348e5808a6aa287db21bf to VOTER
I20260810 19:20:04.253029 19196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } }
I20260810 19:20:04.255733 18547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Refusing update from remote peer 83bb9449eafe4dcd909a5eaf6160fbbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 19:20:04.256018 18863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 LEARNER]: Refusing update from remote peer 83bb9449eafe4dcd909a5eaf6160fbbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 19:20:04.256512 19198 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 19:20:04.257012 19196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 19:20:04.261032 18930 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d5edef1593d348e5808a6aa287db21bf (127.7.43.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.262149 18863 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d5edef1593d348e5808a6aa287db21bf (127.7.43.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.264737 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.265097 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:04.265224 18908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.265091 18547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d5edef1593d348e5808a6aa287db21bf (127.7.43.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.265471 18908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:20:04.265506 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.265748 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:04.266572 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.266885 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:04.269060 19205 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:04.269482 19205 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 19:20:04.491333 19194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:04.491598 19194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:04.491699 19194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:04.499051 19194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:04.499158 19194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:04.499279 19194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:04.509183 19194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:35867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=40483
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:04.510530 19194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:04.511785 19194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:04.524642 19211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:04.524704 19212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:04.524698 19214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:04.525537 19194 server_base.cc:1034] running on GCE node
I20260810 19:20:04.526098 19194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:04.526989 19194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:04.528215 19194 hybrid_clock.cc:648] HybridClock initialized: now 1786389604528162 us; error 42 us; skew 500 ppm
I20260810 19:20:04.528559 19194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389604, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:04.534091 19194 init.cc:377] Logged in from keytab as oryx/127.7.43.2@KRBTEST.COM (short username oryx)
I20260810 19:20:04.535559 19194 webserver.cc:460] Webserver started at http://127.7.43.2:40483/ using document root <none> and password file <none>
I20260810 19:20:04.536521 19194 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:04.536689 19194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:04.542702 19194 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:20:04.545316 19223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:04.546641 19194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:20:04.546789 19194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "fb44bebcde3747aca0b1d06285d1a392"
format_stamp: "Formatted at 2026-08-10 19:20:00 on dist-test-slave-7lbf"
server_key: "2dbfb90013d0b9abce36e7852707f202"
server_key_iv: "8f4a9d74dea1e2568739fdaf0b4fdeca"
server_key_version: "encryptionkey@0"
I20260810 19:20:04.547778 19194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:04.569309 19194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:04.573334 19194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:04.573613 19194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:04.574558 19194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:04.575906 19194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:04.576004 19194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:04.576112 19194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:04.576192 19194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:04.607082 19194 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:35867
I20260810 19:20:04.607146 19432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:35867 every 8 connection(s)
I20260810 19:20:04.608589 19194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:04.618888 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 19194
I20260810 19:20:04.619192 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 18073
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389604, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.60@KRBTEST.COM
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389604, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.3@KRBTEST.COM
Aug 10 19:20:04 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389604, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
I20260810 19:20:04.655979 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.611738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:51981 (local address 127.7.43.60:36351)
0810 19:20:04.622910 (+ 11172us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:04.622918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.629603 (+ 6685us) server_negotiation.cc:407] Connection header received
0810 19:20:04.629785 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.629791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.629894 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.629986 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.634403 (+ 4417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.635796 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.638932 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.639879 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.639982 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.647073 (+ 7091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.647080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.647084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.647134 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.651750 (+ 4616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.652773 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.652778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.652782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.652952 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.653614 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.653619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.653622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.653993 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.654160 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.655279 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.655542 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11015,"thread_start_us":11175,"threads_started":1}
I20260810 19:20:04.658027 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.611659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:38135 (local address 127.7.43.62:38017)
0810 19:20:04.623975 (+ 12316us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:04.623982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.627889 (+ 3907us) server_negotiation.cc:407] Connection header received
0810 19:20:04.629559 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.629565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.629675 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.629798 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.634097 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.635398 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.638932 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.639629 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.639733 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.647056 (+ 7323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.647063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.647067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.647112 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.653675 (+ 6563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.655065 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.655071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.655075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.655335 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.655918 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.655924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.655927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.656383 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.656526 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.657171 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.657407 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12179,"thread_start_us":12115,"threads_started":1}
I20260810 19:20:04.659098 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.612240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.7.43.2:51981)
0810 19:20:04.625726 (+ 13486us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:04.625785 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:04.629648 (+ 3863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:04.630352 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:04.630358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:04.631933 (+ 1575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:04.634160 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.634181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.636100 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.636106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.638763 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.638778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.640437 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.640445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.640664 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.641717 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:04.641760 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:04.646737 (+ 4977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:04.651947 (+ 5210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.651956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.651978 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.652616 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.653116 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.653121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.653124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.653434 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.654266 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:04.654275 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:04.654815 (+ 540us) client_negotiation.cc:769] Sending connection context
0810 19:20:04.656606 (+ 1791us) client_negotiation.cc:240] Negotiation successful
0810 19:20:04.657218 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13093,"mutex_wait_us":150,"thread_start_us":11230,"threads_started":1}
I20260810 19:20:04.659826 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.625200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:40267 (local address 127.7.43.61:41565)
0810 19:20:04.627975 (+ 2775us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:04.627981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.634592 (+ 6611us) server_negotiation.cc:407] Connection header received
0810 19:20:04.634744 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.634750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.634883 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.634982 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.637238 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.638694 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.643485 (+ 4791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.644366 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.644505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.650029 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.650037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.650042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.650130 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.655067 (+ 4937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.655982 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.655988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.655991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.656187 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.656820 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.656826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.656829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.657174 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.657321 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.659053 (+ 1732us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.659290 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2659,"thread_start_us":2524,"threads_started":1}
I20260810 19:20:04.660154 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.620738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.7.43.3:39199)
0810 19:20:04.626320 (+ 5582us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:04.626328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:04.628137 (+ 1809us) server_negotiation.cc:407] Connection header received
0810 19:20:04.628269 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:04.628279 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:04.628556 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:04.628763 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:04.631586 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.633570 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.636293 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.637593 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.637735 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.648533 (+ 10798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:04.648557 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:04.648578 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:04.648630 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:04.654809 (+ 6179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.656077 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.656088 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.656092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.656323 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:04.656905 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:04.656910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:04.656913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:04.657340 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:04.657659 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:04.658986 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0810 19:20:04.659269 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5394,"thread_start_us":5200,"threads_started":1}
I20260810 19:20:04.660208 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.625090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.7.43.2:40267)
0810 19:20:04.634481 (+ 9391us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:04.634496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:04.634605 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:04.635236 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:04.635241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:04.636152 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:04.636869 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.636886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.639036 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.639041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.643282 (+ 4241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.643298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.644615 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.644625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.644958 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.645797 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:04.645828 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:04.649672 (+ 3844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:04.655225 (+ 5553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.655230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.655233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.655850 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.656383 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.656389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.656392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.656685 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.657386 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:04.657390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:04.657588 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 19:20:04.657728 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 19:20:04.658062 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9293,"mutex_wait_us":82,"thread_start_us":5284,"threads_started":1}
I20260810 19:20:04.660552 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:04.612240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:38135)
0810 19:20:04.627455 (+ 15215us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:04.627476 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:04.629162 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:04.630265 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:04.630278 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:04.632469 (+ 2191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:04.633869 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.633902 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.635628 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.635633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.638765 (+ 3132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:04.638778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:04.640139 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:04.640149 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:04.640543 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:04.641733 (+ 1190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:04.641824 (+ 91us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:04.646737 (+ 4913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:04.653919 (+ 7182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.653924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.653928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.654894 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.655499 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:04.655504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:04.655507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:04.655753 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:04.656612 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:04.656617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:04.656761 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 19:20:04.658504 (+ 1743us) client_negotiation.cc:240] Negotiation successful
0810 19:20:04.658776 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15017,"mutex_wait_us":535,"spinlock_wait_cycles":24960,"thread_start_us":21671,"threads_started":1}
I20260810 19:20:04.661266 19433 heartbeater.cc:347] Connected to a master server at 127.7.43.60:36351
I20260810 19:20:04.661654 19434 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38017
I20260810 19:20:04.661826 19434 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:04.661891 18282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:04.662849 19434 heartbeater.cc:511] Master 127.7.43.62:38017 requested a full tablet report, sending...
I20260810 19:20:04.663731 19435 heartbeater.cc:347] Connected to a master server at 127.7.43.61:41565
I20260810 19:20:04.663859 19435 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:04.664510 19435 heartbeater.cc:511] Master 127.7.43.61:41565 requested a full tablet report, sending...
I20260810 19:20:04.661661 19433 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:04.666159 19433 heartbeater.cc:511] Master 127.7.43.60:36351 requested a full tablet report, sending...
I20260810 19:20:04.666146 18537 ts_manager.cc:194] Registered new tserver with Master: fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867)
I20260810 19:20:04.666153 17543 ts_manager.cc:194] Re-registered known tserver with Master: fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867)
I20260810 19:20:04.667484 17543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:38135
I20260810 19:20:04.667793 18850 ts_manager.cc:194] Registered new tserver with Master: fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867)
I20260810 19:20:04.755707 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 18073
I20260810 19:20:04.783326 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:39199
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:38017
--webserver_port=32909
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:20:05.039749 19176 heartbeater.cc:503] Master 127.7.43.62:38017 was elected leader, sending a full tablet report...
W20260810 19:20:05.134353 19454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:05.134657 19454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:05.134755 19454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:05.142150 19454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:05.142264 19454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:05.142385 19454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:05.152294 19454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:39199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=32909
--tserver_master_addrs=127.7.43.62:38017,127.7.43.61:41565,127.7.43.60:36351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:05.153642 19454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:05.154933 19454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:05.168598 19462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:05.168639 19469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:05.168669 19464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:05.169395 19454 server_base.cc:1034] running on GCE node
I20260810 19:20:05.169996 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:05.170962 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:05.172156 19454 hybrid_clock.cc:648] HybridClock initialized: now 1786389605172090 us; error 61 us; skew 500 ppm
I20260810 19:20:05.172483 19454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:05 dist-test-slave-7lbf krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389605, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:05.175514 19454 init.cc:377] Logged in from keytab as oryx/127.7.43.3@KRBTEST.COM (short username oryx)
I20260810 19:20:05.176867 19454 webserver.cc:460] Webserver started at http://127.7.43.3:32909/ using document root <none> and password file <none>
I20260810 19:20:05.177588 19454 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:05.177723 19454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:05.183382 19454 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:20:05.186132 19479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:05.187430 19454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:20:05.187564 19454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "bebf7e6e27244a969ae4a741d2d4093a"
format_stamp: "Formatted at 2026-08-10 19:20:01 on dist-test-slave-7lbf"
server_key: "4886a6ebe0609354f4857d9b39fbafe4"
server_key_iv: "1b817544b7d9b8f307128b42e85f603e"
server_key_version: "encryptionkey@0"
I20260810 19:20:05.188513 19454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:05.211679 19454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:05.215606 19454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:05.215839 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:05.216823 19454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:05.218298 19454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:05.218392 19454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:05.218482 19454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:05.218536 19454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:05.249243 19454 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:39199
I20260810 19:20:05.249300 19671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:39199 every 8 connection(s)
I20260810 19:20:05.250820 19454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:05.254333 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 19454
Aug 10 19:20:05 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389605, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
Aug 10 19:20:05 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389605, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.60@KRBTEST.COM
Aug 10 19:20:05 dist-test-slave-7lbf krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389605, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.61@KRBTEST.COM
I20260810 19:20:05.317564 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.253662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:48867 (local address 127.7.43.62:38017)
0810 19:20:05.279676 (+ 26014us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.279683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.280042 (+ 359us) server_negotiation.cc:407] Connection header received
0810 19:20:05.282766 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.282772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.282854 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.282966 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.287555 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.288677 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.295992 (+ 7315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.296586 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.296696 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.301349 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.301357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.301361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.301404 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.308876 (+ 7472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.310669 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.310674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.310677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.311168 (+ 491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.312368 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.312374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.312377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.313080 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.314077 (+ 997us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.316612 (+ 2535us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.316915 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25899,"thread_start_us":43164,"threads_started":1}
I20260810 19:20:05.318961 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.253625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:50059 (local address 127.7.43.60:36351)
0810 19:20:05.274178 (+ 20553us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.274186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.278663 (+ 4477us) server_negotiation.cc:407] Connection header received
0810 19:20:05.282990 (+ 4327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.282995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.283082 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.283164 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.287603 (+ 4439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.288677 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.294815 (+ 6138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.295346 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.295446 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.302129 (+ 6683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.302136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.302140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.302181 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.307964 (+ 5783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.311338 (+ 3374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.311344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.311348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.311558 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.314084 (+ 2526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.314090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.314093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.314485 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.314621 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.318034 (+ 3413us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.318379 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20420,"thread_start_us":20301,"threads_started":1}
I20260810 19:20:05.321583 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.254463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.7.43.3:50059)
0810 19:20:05.278454 (+ 23991us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.278508 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.282836 (+ 4328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.283666 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.283680 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.284747 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:05.285695 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.286404 (+ 709us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.288899 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.288905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.294669 (+ 5764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.294682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.295758 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.295764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.296033 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.299211 (+ 3178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:05.299234 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:05.301894 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:05.309039 (+ 7145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:05.309043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:05.309060 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:05.309977 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:05.313707 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:05.313713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:05.313717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:05.313949 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:05.314629 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:05.314638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:05.315226 (+ 588us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.315616 (+ 390us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.316215 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22566,"mutex_wait_us":82,"spinlock_wait_cycles":37376,"thread_start_us":31145,"threads_started":1}
I20260810 19:20:05.321704 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.255211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:48867)
0810 19:20:05.279954 (+ 24743us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.279973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.282456 (+ 2483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.286691 (+ 4235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.286696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.287141 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:05.287404 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.287415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.292384 (+ 4969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.292390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.294074 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.294088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.297048 (+ 2960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.297052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.297171 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.297927 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:05.297960 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:05.301110 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:05.309029 (+ 7919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:05.309039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:05.309060 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:05.310508 (+ 1448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:05.311714 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:05.311719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:05.311722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:05.312213 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:05.314666 (+ 2453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:05.314671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:05.315226 (+ 555us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.320628 (+ 5402us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.320959 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24624,"mutex_wait_us":12,"spinlock_wait_cycles":1408,"thread_start_us":22042,"threads_started":1}
I20260810 19:20:05.323625 19672 heartbeater.cc:347] Connected to a master server at 127.7.43.60:36351
I20260810 19:20:05.324043 19672 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:05.325456 19672 heartbeater.cc:511] Master 127.7.43.60:36351 requested a full tablet report, sending...
I20260810 19:20:05.325599 19674 heartbeater.cc:347] Connected to a master server at 127.7.43.62:38017
I20260810 19:20:05.325778 19674 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:05.327099 19674 heartbeater.cc:511] Master 127.7.43.62:38017 requested a full tablet report, sending...
I20260810 19:20:05.327934 18850 ts_manager.cc:194] Registered new tserver with Master: bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199)
I20260810 19:20:05.328150 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.286206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.7.43.3:32977)
0810 19:20:05.290085 (+ 3879us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.290104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.292498 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.304331 (+ 11833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.304337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.304969 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:05.305308 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.305320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.307107 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.307113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.308918 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.308932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.310262 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.310272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.310558 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.311150 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:05.311173 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:05.314823 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:05.320139 (+ 5316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:05.320145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:05.320148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:05.320659 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:05.325102 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:05.325107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:05.325111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:05.325400 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:05.326819 (+ 1419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:05.326824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:05.327001 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.327153 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.327523 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3754,"thread_start_us":4034,"threads_started":1}
I20260810 19:20:05.330049 17543 ts_manager.cc:194] Re-registered known tserver with Master: bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199)
I20260810 19:20:05.331164 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.286303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:32977 (local address 127.7.43.61:41565)
0810 19:20:05.304020 (+ 17717us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.304026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.304043 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:20:05.304086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.304091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.304149 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.305470 (+ 1321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.305686 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.306824 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.309116 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.309938 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.310882 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.315072 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.315080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.315085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.315124 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.320001 (+ 4877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.324680 (+ 4679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.324686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.324689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.324941 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.325588 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.325593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.325596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.326409 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.329316 (+ 2907us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.330186 (+ 870us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.330491 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17596,"thread_start_us":17323,"threads_started":1}
I20260810 19:20:05.331441 17543 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:48867
I20260810 19:20:05.332352 19677 heartbeater.cc:347] Connected to a master server at 127.7.43.61:41565
I20260810 19:20:05.332537 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.279233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.7.43.62:38017)
0810 19:20:05.294502 (+ 15269us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.294507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.296573 (+ 2066us) server_negotiation.cc:407] Connection header received
0810 19:20:05.296627 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.296631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.296708 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.296815 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.305322 (+ 8507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.306692 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.309373 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.310286 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.310406 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.318027 (+ 7621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.318033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.318037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.318072 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.324533 (+ 6461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.325874 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.325879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.325882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.326085 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.328375 (+ 2290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.328380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.328384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.328839 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.329203 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.329628 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.329922 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15148,"thread_start_us":228,"threads_started":1}
I20260810 19:20:05.332566 19677 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:05.333639 19677 heartbeater.cc:511] Master 127.7.43.61:41565 requested a full tablet report, sending...
I20260810 19:20:05.335377 18537 ts_manager.cc:194] Registered new tserver with Master: bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199)
I20260810 19:20:05.339087 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.296012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.7.43.60:36351)
0810 19:20:05.307928 (+ 11916us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.307934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.307954 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:20:05.308046 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.308051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.308153 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.308257 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.315748 (+ 7491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.317641 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.323472 (+ 5831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.324423 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.324552 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.331389 (+ 6837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.331399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.331405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.331463 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.335379 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.336486 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.336494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.336498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.336750 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.337242 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.337249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.337252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.337690 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.337942 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.338302 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.338544 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11787,"thread_start_us":13030,"threads_started":1}
I20260810 19:20:05.341862 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.279710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.7.43.61:41565)
0810 19:20:05.303194 (+ 23484us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.303201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.303225 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:20:05.303292 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.303298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.303392 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.303508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.310099 (+ 6591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.311691 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.318021 (+ 6330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.319191 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.319297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.329449 (+ 10152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.329459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.329463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.329544 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.338138 (+ 8594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.339070 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.339076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.339079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.339257 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.339724 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.339730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.339733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.340309 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.340596 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.341211 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.341451 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23340,"thread_start_us":27215,"threads_started":1}
I20260810 19:20:05.338888 17543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:05.350796 17543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:05.394104 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.375579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39199 (local address 127.0.0.1:54050)
0810 19:20:05.382512 (+ 6933us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.382540 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.382724 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.383775 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.383780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.383806 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:05.384676 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.384690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.387505 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.387509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.390219 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.390235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.392949 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.392959 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.393148 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.393214 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.393329 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.393494 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6717,"mutex_wait_us":69,"thread_start_us":9079,"threads_started":1}
I20260810 19:20:05.394210 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.375941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.7.43.3:39199)
0810 19:20:05.377449 (+ 1508us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.377459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.382668 (+ 5209us) server_negotiation.cc:407] Connection header received
0810 19:20:05.382954 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.382964 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.383293 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.383573 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:05.385024 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.387189 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.390426 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.392683 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.392818 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.393078 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.393412 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.393641 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":893,"threads_started":1}
I20260810 19:20:05.396034 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.377128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.7.43.2:35867)
0810 19:20:05.382360 (+ 5232us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.382370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.383694 (+ 1324us) server_negotiation.cc:407] Connection header received
0810 19:20:05.384676 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.384685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.385037 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.385191 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:05.386318 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.388069 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.392132 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.394199 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.394329 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.394608 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.395371 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.395602 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5043,"thread_start_us":4871,"threads_started":1}
I20260810 19:20:05.396064 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.376574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35867 (local address 127.0.0.1:41218)
0810 19:20:05.383553 (+ 6979us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.383594 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.384446 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.385468 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.385471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.385492 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:05.386117 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.386129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.388261 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.388267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.391942 (+ 3675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.391960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.394871 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.394880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.395128 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.395190 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.395276 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.395398 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6831,"mutex_wait_us":83,"thread_start_us":5499,"threads_started":1}
I20260810 19:20:05.398886 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.376594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:36603 (local address 127.0.0.1:59776)
0810 19:20:05.386467 (+ 9873us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.386499 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.386639 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.388678 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.388684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.388710 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:05.389934 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.389956 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.392150 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.392155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.395197 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.395210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.398021 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.398026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.398221 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.398244 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.398315 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.398461 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9756,"mutex_wait_us":53,"thread_start_us":6570,"threads_started":1}
I20260810 19:20:05.399108 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.377741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.7.43.1:36603)
0810 19:20:05.378691 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.378700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.386598 (+ 7898us) server_negotiation.cc:407] Connection header received
0810 19:20:05.386788 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.386797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.387031 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.387219 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:05.390268 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.391957 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.395374 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.397363 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.397473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.397800 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.398409 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.398601 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":550,"threads_started":1}
I20260810 19:20:05.399008 19557 tablet_service.cc:1467] Processing CreateTablet for tablet 24f805cd6fdf401b809a3f6bd53cf3c0 (DEFAULT_TABLE table=test-table [id=6f17a40a4ba44345ae41dc65fc61f453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:05.400686 19557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f805cd6fdf401b809a3f6bd53cf3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:05.403568 19045 tablet_service.cc:1467] Processing CreateTablet for tablet 24f805cd6fdf401b809a3f6bd53cf3c0 (DEFAULT_TABLE table=test-table [id=6f17a40a4ba44345ae41dc65fc61f453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:05.405468 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f805cd6fdf401b809a3f6bd53cf3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:05.408855 19316 tablet_service.cc:1467] Processing CreateTablet for tablet 24f805cd6fdf401b809a3f6bd53cf3c0 (DEFAULT_TABLE table=test-table [id=6f17a40a4ba44345ae41dc65fc61f453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:05.410671 19316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f805cd6fdf401b809a3f6bd53cf3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:05.423269 19734 tablet_bootstrap.cc:492] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Bootstrap starting.
I20260810 19:20:05.426575 19737 tablet_bootstrap.cc:492] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Bootstrap starting.
I20260810 19:20:05.428160 19734 tablet_bootstrap.cc:654] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:05.431578 19734 log.cc:824] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:05.433071 19737 tablet_bootstrap.cc:654] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:05.435731 19737 log.cc:824] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:05.437728 19738 tablet_bootstrap.cc:492] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Bootstrap starting.
I20260810 19:20:05.443207 19734 tablet_bootstrap.cc:492] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: No bootstrap required, opened a new log
I20260810 19:20:05.443714 19734 ts_tablet_manager.cc:1397] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20260810 19:20:05.444730 19738 tablet_bootstrap.cc:654] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:05.446363 19737 tablet_bootstrap.cc:492] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: No bootstrap required, opened a new log
I20260810 19:20:05.446710 19737 ts_tablet_manager.cc:1397] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20260810 19:20:05.446864 19738 log.cc:824] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:05.459631 19734 raft_consensus.cc:348] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.460294 19734 raft_consensus.cc:374] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:05.460384 19738 tablet_bootstrap.cc:492] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: No bootstrap required, opened a new log
I20260810 19:20:05.460458 19734 raft_consensus.cc:729] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebf7e6e27244a969ae4a741d2d4093a, State: Initialized, Role: FOLLOWER
I20260810 19:20:05.460911 19738 ts_tablet_manager.cc:1397] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.008s
I20260810 19:20:05.461194 19734 consensus_queue.cc:260] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.461984 19737 raft_consensus.cc:348] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.462761 19737 raft_consensus.cc:374] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:05.462867 19674 heartbeater.cc:503] Master 127.7.43.62:38017 was elected leader, sending a full tablet report...
I20260810 19:20:05.462945 19737 raft_consensus.cc:729] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64aaeddbe424cea978f4db16dfd3778, State: Initialized, Role: FOLLOWER
I20260810 19:20:05.463682 19734 ts_tablet_manager.cc:1428] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
I20260810 19:20:05.463800 19737 consensus_queue.cc:260] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.470108 19737 ts_tablet_manager.cc:1428] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Time spent starting tablet: real 0.023s user 0.018s sys 0.004s
I20260810 19:20:05.480139 19738 raft_consensus.cc:348] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.480762 19738 raft_consensus.cc:374] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:05.480911 19738 raft_consensus.cc:729] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb44bebcde3747aca0b1d06285d1a392, State: Initialized, Role: FOLLOWER
I20260810 19:20:05.481670 19738 consensus_queue.cc:260] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.483446 19434 heartbeater.cc:503] Master 127.7.43.62:38017 was elected leader, sending a full tablet report...
I20260810 19:20:05.483747 19738 ts_tablet_manager.cc:1428] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Time spent starting tablet: real 0.023s user 0.006s sys 0.010s
W20260810 19:20:05.490625 19180 tablet.cc:2369] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:05.515044 19746 raft_consensus.cc:482] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:05.515453 19746 raft_consensus.cc:504] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.517460 19746 leader_election.cc:288] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199), c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603)
W20260810 19:20:05.521111 19679 tablet.cc:2369] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:05.527869 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.517929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:34099 (local address 127.7.43.1:36603)
0810 19:20:05.518191 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.518198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.519535 (+ 1337us) server_negotiation.cc:407] Connection header received
0810 19:20:05.519840 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.519845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.519910 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.519986 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:05.521048 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.522442 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.525761 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.526887 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.527007 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.527126 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.527441 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.527607 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 19:20:05.528151 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.517719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:36603 (local address 127.7.43.2:34099)
0810 19:20:05.519406 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.519423 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.519615 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.520029 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.520034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.520061 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:05.520860 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.520876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.522626 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.522630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.525597 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.525615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.527140 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.527145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.527287 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.527313 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.527387 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.527485 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"mutex_wait_us":1,"thread_start_us":1051,"threads_started":1}
I20260810 19:20:05.529059 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.517928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:39297 (local address 127.7.43.3:39199)
0810 19:20:05.518200 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.518205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.519911 (+ 1706us) server_negotiation.cc:407] Connection header received
0810 19:20:05.520104 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.520108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.520170 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.520244 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:05.521355 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.522775 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.526275 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.527872 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.528008 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.528109 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.528676 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.528776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:20:05.529235 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.518128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39199 (local address 127.7.43.2:39297)
0810 19:20:05.519801 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:05.519816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:05.519973 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:05.520280 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:05.520284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:05.520314 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:05.521144 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.521160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.522940 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.522946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.526093 (+ 3147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:05.526105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.528282 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.528289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:05.528505 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.528543 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:20:05.528643 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:20:05.528761 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"mutex_wait_us":22,"thread_start_us":1350,"threads_started":1}
I20260810 19:20:05.529549 19087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f805cd6fdf401b809a3f6bd53cf3c0" candidate_uuid: "fb44bebcde3747aca0b1d06285d1a392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64aaeddbe424cea978f4db16dfd3778" is_pre_election: true
I20260810 19:20:05.530280 19087 raft_consensus.cc:2454] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb44bebcde3747aca0b1d06285d1a392 in term 0.
I20260810 19:20:05.530417 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f805cd6fdf401b809a3f6bd53cf3c0" candidate_uuid: "fb44bebcde3747aca0b1d06285d1a392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebf7e6e27244a969ae4a741d2d4093a" is_pre_election: true
I20260810 19:20:05.531251 19230 leader_election.cc:302] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64aaeddbe424cea978f4db16dfd3778, fb44bebcde3747aca0b1d06285d1a392; no voters:
I20260810 19:20:05.531248 19593 raft_consensus.cc:2454] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb44bebcde3747aca0b1d06285d1a392 in term 0.
I20260810 19:20:05.531850 19746 raft_consensus.cc:2781] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:05.532100 19746 raft_consensus.cc:482] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:05.532279 19746 raft_consensus.cc:3037] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:05.535956 19746 raft_consensus.cc:504] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.536813 19746 leader_election.cc:288] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [CANDIDATE]: Term 1 election: Requested vote from peers bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199), c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603)
I20260810 19:20:05.537150 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f805cd6fdf401b809a3f6bd53cf3c0" candidate_uuid: "fb44bebcde3747aca0b1d06285d1a392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebf7e6e27244a969ae4a741d2d4093a"
I20260810 19:20:05.537285 19087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f805cd6fdf401b809a3f6bd53cf3c0" candidate_uuid: "fb44bebcde3747aca0b1d06285d1a392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64aaeddbe424cea978f4db16dfd3778"
I20260810 19:20:05.537354 19593 raft_consensus.cc:3037] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:05.537540 19087 raft_consensus.cc:3037] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:05.540154 19593 raft_consensus.cc:2454] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb44bebcde3747aca0b1d06285d1a392 in term 1.
I20260810 19:20:05.540555 19224 leader_election.cc:302] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bebf7e6e27244a969ae4a741d2d4093a, fb44bebcde3747aca0b1d06285d1a392; no voters:
I20260810 19:20:05.540637 19087 raft_consensus.cc:2454] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb44bebcde3747aca0b1d06285d1a392 in term 1.
I20260810 19:20:05.540927 19746 raft_consensus.cc:2781] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:05.541832 19746 raft_consensus.cc:686] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 1 LEADER]: Becoming Leader. State: Replica: fb44bebcde3747aca0b1d06285d1a392, State: Running, Role: LEADER
I20260810 19:20:05.542510 19746 consensus_queue.cc:237] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } }
I20260810 19:20:05.547727 17543 catalog_manager.cc:5495] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "fb44bebcde3747aca0b1d06285d1a392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } health_report { overall_health: HEALTHY } } }
I20260810 19:20:05.586846 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.580362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.7.43.2:35867)
0810 19:20:05.580564 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.580569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.580585 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:20:05.580671 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.580675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.580793 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.580922 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:05.581570 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.582571 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.584312 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.585000 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.585093 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.585603 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:05.586245 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:05.586355 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.586465 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.586531 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:20:05.597769 19087 raft_consensus.cc:1261] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Refusing update from remote peer fb44bebcde3747aca0b1d06285d1a392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:05.597749 19593 raft_consensus.cc:1261] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Refusing update from remote peer fb44bebcde3747aca0b1d06285d1a392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:05.599069 19746 consensus_queue.cc:1035] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:05.599536 19753 consensus_queue.cc:1035] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:20:05.612304 19436 tablet.cc:2369] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:05.619385 19763 mvcc.cc:205] Tried to move back new op lower bound from 7317051824516694016 to 7317051824306585600. Current Snapshot: MvccSnapshot[applied={T|T < 7317051824516694016}]
I20260810 19:20:05.659379 17543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:05.659860 17543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:05.667050 17543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83bb9449eafe4dcd909a5eaf6160fbbf: Sending DeleteTablet for 3 replicas of tablet 24f805cd6fdf401b809a3f6bd53cf3c0
I20260810 19:20:05.668507 19557 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f805cd6fdf401b809a3f6bd53cf3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:05 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:54050
I20260810 19:20:05.669138 19316 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f805cd6fdf401b809a3f6bd53cf3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:05 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:41218
I20260810 19:20:05.670905 19045 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f805cd6fdf401b809a3f6bd53cf3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:05 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59776
I20260810 19:20:05.671965 19780 tablet_replica.cc:330] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: stopping tablet replica
I20260810 19:20:05.672564 19782 tablet_replica.cc:330] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: stopping tablet replica
I20260810 19:20:05.672521 19780 raft_consensus.cc:2229] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:05.673238 19780 raft_consensus.cc:2258] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:05.673096 19782 raft_consensus.cc:2229] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:05.673902 19782 raft_consensus.cc:2258] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:05.675732 19780 ts_tablet_manager.cc:1905] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:05.676011 19782 ts_tablet_manager.cc:1905] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:05.679263 19781 tablet_replica.cc:330] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: stopping tablet replica
I20260810 19:20:05.679836 19781 raft_consensus.cc:2229] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:05.680985 19781 raft_consensus.cc:2258] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:05.683920 19781 ts_tablet_manager.cc:1905] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:05.684506 19782 ts_tablet_manager.cc:1918] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:05.684804 19782 log.cc:1196] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/24f805cd6fdf401b809a3f6bd53cf3c0
I20260810 19:20:05.685854 19782 ts_tablet_manager.cc:1939] T 24f805cd6fdf401b809a3f6bd53cf3c0 P c64aaeddbe424cea978f4db16dfd3778: Deleting consensus metadata
I20260810 19:20:05.688133 17523 catalog_manager.cc:4841] TS c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603): tablet 24f805cd6fdf401b809a3f6bd53cf3c0 (table test-table [id=6f17a40a4ba44345ae41dc65fc61f453]) successfully deleted
I20260810 19:20:05.690217 19780 ts_tablet_manager.cc:1918] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:05.690555 19780 log.cc:1196] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/24f805cd6fdf401b809a3f6bd53cf3c0
I20260810 19:20:05.690649 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 17491
I20260810 19:20:05.691340 19780 ts_tablet_manager.cc:1939] T 24f805cd6fdf401b809a3f6bd53cf3c0 P bebf7e6e27244a969ae4a741d2d4093a: Deleting consensus metadata
I20260810 19:20:05.691985 17594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:05.702031 19781 ts_tablet_manager.cc:1918] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:05.702462 19781 log.cc:1196] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/24f805cd6fdf401b809a3f6bd53cf3c0
I20260810 19:20:05.703671 19781 ts_tablet_manager.cc:1939] T 24f805cd6fdf401b809a3f6bd53cf3c0 P fb44bebcde3747aca0b1d06285d1a392: Deleting consensus metadata
I20260810 19:20:05.714252 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.674322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.7.43.61:41565)
0810 19:20:05.674501 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.674507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.688323 (+ 13816us) server_negotiation.cc:407] Connection header received
0810 19:20:05.688599 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.688605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.688694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.688799 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.692668 (+ 3869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.694286 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.696571 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.697655 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.697789 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.703264 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.703272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.703278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.703466 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.709476 (+ 6010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.710512 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.710519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.710523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.710772 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.711657 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.711675 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.711679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.712213 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.712466 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.713210 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.713491 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:20:05.731653 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.689370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.7.43.60:36351)
0810 19:20:05.689696 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.689702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.713920 (+ 24218us) server_negotiation.cc:407] Connection header received
0810 19:20:05.714235 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.714244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.714346 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.714449 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.717338 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.718535 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.720416 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.720970 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.721090 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.724743 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.724750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.724753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.724787 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.728325 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.729207 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.729213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.729217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.729390 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.729858 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.729862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.729864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.730233 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.730570 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.730962 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.731231 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:20:05.866211 17520 catalog_manager.cc:4841] TS bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199): tablet 24f805cd6fdf401b809a3f6bd53cf3c0 (table test-table [id=6f17a40a4ba44345ae41dc65fc61f453]) successfully deleted
I20260810 19:20:05.866640 17521 catalog_manager.cc:4841] TS fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867): tablet 24f805cd6fdf401b809a3f6bd53cf3c0 (table test-table [id=6f17a40a4ba44345ae41dc65fc61f453]) successfully deleted
W20260810 19:20:05.866533 17520 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.43.3:39199, user_credentials={real_user=slave}} blocked reactor thread for 173157us
W20260810 19:20:05.866863 17521 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.43.2:35867, user_credentials={real_user=slave}} blocked reactor thread for 160838us
I20260810 19:20:05.867414 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 17491
I20260810 19:20:05.918689 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.903380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.7.43.60:36351)
0810 19:20:05.903597 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.903603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.903623 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:20:05.903688 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.903694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.903776 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.903891 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.906207 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.907001 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.908534 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.909202 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.909332 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.912787 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.912794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.912797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.912835 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.915595 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.916376 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.916381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.916385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.916538 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.917001 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.917008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.917012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.917434 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.917697 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.918055 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.918312 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:20:05.920185 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:05.900876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.7.43.61:41565)
0810 19:20:05.901116 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:05.901124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:05.902604 (+ 1480us) server_negotiation.cc:407] Connection header received
0810 19:20:05.902935 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:05.902942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:05.903020 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:05.903126 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:05.905169 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.906252 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.908237 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:05.908769 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:05.908908 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:05.912506 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:05.912513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:05.912519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:05.912564 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:05.916102 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.916857 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.916862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.916865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.917059 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:05.917487 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:05.917492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:05.917495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:05.917899 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:05.918154 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:05.919581 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0810 19:20:05.919855 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:20:06.634196 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:06.633224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:47895)
0810 19:20:06.633761 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:06.633923 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
W20260810 19:20:06.634723 19674 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:38017 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:06.643610 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:06.642747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:57915)
0810 19:20:06.643243 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:06.643390 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":270,"threads_started":1}
W20260810 19:20:06.644021 19434 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:38017 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:06.655762 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:06.654861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:39857)
0810 19:20:06.655392 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:06.655549 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":253,"threads_started":1}
W20260810 19:20:06.656147 19176 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:38017 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:07.168494 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83bb9449eafe4dcd909a5eaf6160fbbf)
I20260810 19:20:07.168860 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } }
I20260810 19:20:07.170300 19802 leader_election.cc:288] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017), d5edef1593d348e5808a6aa287db21bf (127.7.43.60:36351)
I20260810 19:20:07.172008 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.170616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:36998)
0810 19:20:07.171577 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.171711 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":763,"threads_started":1}
W20260810 19:20:07.172389 18514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:07.174113 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.173695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37006)
0810 19:20:07.173857 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.173956 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 19:20:07.174319 18514 leader_election.cc:334] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017): Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:07.176314 19811 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83bb9449eafe4dcd909a5eaf6160fbbf)
I20260810 19:20:07.176630 19811 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } }
I20260810 19:20:07.177426 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.170556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:36351 (local address 127.0.0.1:60914)
0810 19:20:07.171139 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.171167 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.171320 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.171766 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.171771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.171805 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.172230 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.172246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.173625 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.173629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.175257 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.175270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.176716 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.176720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.176849 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.176875 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.176976 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.177104 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":253,"threads_started":1}
I20260810 19:20:07.177603 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.170826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.7.43.60:36351)
0810 19:20:07.171453 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.171459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.171480 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:20:07.171559 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.171565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.171625 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.171711 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.172439 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.173438 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.175408 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.176538 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.176680 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.176786 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.177047 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.177186 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":307,"threads_started":1}
I20260810 19:20:07.178287 19811 leader_election.cc:288] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017), f77cf2ae35eb4c8b986a5f5979d78074 (127.7.43.61:41565)
I20260810 19:20:07.178294 18863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d5edef1593d348e5808a6aa287db21bf" is_pre_election: true
I20260810 19:20:07.178757 18863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f77cf2ae35eb4c8b986a5f5979d78074 in term 1.
I20260810 19:20:07.179518 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.178395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37018)
0810 19:20:07.179062 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.179258 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":369,"threads_started":1}
W20260810 19:20:07.179814 18827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:07.180436 18517 leader_election.cc:302] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5edef1593d348e5808a6aa287db21bf, f77cf2ae35eb4c8b986a5f5979d78074; no voters: 83bb9449eafe4dcd909a5eaf6160fbbf
I20260810 19:20:07.180878 19802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 19:20:07.181012 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83bb9449eafe4dcd909a5eaf6160fbbf)
I20260810 19:20:07.181108 19802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:20:07.183984 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.180914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37030)
0810 19:20:07.183606 (+ 2692us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.183774 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2561}
I20260810 19:20:07.183795 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } }
W20260810 19:20:07.184149 18827 leader_election.cc:334] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017): Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:07.184581 19802 leader_election.cc:288] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [CANDIDATE]: Term 2 election: Requested vote from peers 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017), d5edef1593d348e5808a6aa287db21bf (127.7.43.60:36351)
I20260810 19:20:07.185070 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.184635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37046)
0810 19:20:07.184858 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.184941 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 19:20:07.185175 18863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d5edef1593d348e5808a6aa287db21bf"
I20260810 19:20:07.185423 18863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:20:07.186040 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.185652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37060)
0810 19:20:07.185787 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.185870 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:20:07.187186 18514 leader_election.cc:334] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017): Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
I20260810 19:20:07.187793 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.178733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.7.43.61:41565)
0810 19:20:07.180442 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.180447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.180471 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:20:07.180530 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.180535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.180610 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.181289 (+ 679us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.181556 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.182991 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.185879 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.186930 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.187024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.187103 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.187463 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.187558 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1561,"thread_start_us":1380,"threads_started":1}
I20260810 19:20:07.187841 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.178515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:41565 (local address 127.0.0.1:54066)
0810 19:20:07.179767 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.179784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.179934 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.180771 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.180776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.180805 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.181135 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.181150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.183168 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.183175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.185677 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.185692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.187107 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.187113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.187309 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.187344 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.187430 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.187566 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":4810,"threads_started":1}
I20260810 19:20:07.188216 18863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f77cf2ae35eb4c8b986a5f5979d78074 in term 2.
I20260810 19:20:07.188438 18547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5edef1593d348e5808a6aa287db21bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" is_pre_election: true
I20260810 19:20:07.188905 18547 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5edef1593d348e5808a6aa287db21bf in current term 2: Already voted for candidate f77cf2ae35eb4c8b986a5f5979d78074 in this term.
I20260810 19:20:07.189065 18517 leader_election.cc:302] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d5edef1593d348e5808a6aa287db21bf, f77cf2ae35eb4c8b986a5f5979d78074; no voters: 83bb9449eafe4dcd909a5eaf6160fbbf
I20260810 19:20:07.189316 19802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:20:07.189730 18828 leader_election.cc:302] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5edef1593d348e5808a6aa287db21bf; no voters: 83bb9449eafe4dcd909a5eaf6160fbbf, f77cf2ae35eb4c8b986a5f5979d78074
I20260810 19:20:07.190011 19802 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [term 2 LEADER]: Becoming Leader. State: Replica: f77cf2ae35eb4c8b986a5f5979d78074, State: Running, Role: LEADER
I20260810 19:20:07.190097 19811 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:20:07.190475 19802 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } }
I20260810 19:20:07.192744 19816 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77cf2ae35eb4c8b986a5f5979d78074. Latest consensus state: current_term: 2 leader_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:07.193185 19816 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:07.193858 19819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:07.198096 19819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:07.198926 19819 catalog_manager.cc:1259] Loaded cluster ID: 84754b7befde449a98e32871b6cfdf53
I20260810 19:20:07.198997 19819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:07.200135 19819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:07.200935 19819 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Loaded TSK: 0
I20260810 19:20:07.201889 19819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:20:07.252784 18537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:07.259878 18537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:07.263978 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.263574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37102)
0810 19:20:07.263734 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.263825 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:20:07.264104 18863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [term 2 FOLLOWER]: Refusing update from remote peer f77cf2ae35eb4c8b986a5f5979d78074: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 19:20:07.264783 19816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 19:20:07.264935 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.264556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37110)
0810 19:20:07.264714 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.264783 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:20:07.265131 18514 consensus_peers.cc:487] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 -> Peer 83bb9449eafe4dcd909a5eaf6160fbbf (127.7.43.62:38017): Couldn't send request to peer 83bb9449eafe4dcd909a5eaf6160fbbf. Status: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 19:20:07.267639 19811 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77cf2ae35eb4c8b986a5f5979d78074. Latest consensus state: current_term: 2 leader_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:07.267923 19811 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:20:07.269361 19816 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:07.269665 19816 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:07.269833 19811 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "83bb9449eafe4dcd909a5eaf6160fbbf" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 38017 } } peers { permanent_uuid: "f77cf2ae35eb4c8b986a5f5979d78074" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 41565 } attrs { promote: false } } peers { permanent_uuid: "d5edef1593d348e5808a6aa287db21bf" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 36351 } attrs { promote: false } } }
I20260810 19:20:07.270138 19811 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5edef1593d348e5808a6aa287db21bf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:20:07.288915 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.278069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39199 (local address 127.0.0.1:54066)
0810 19:20:07.278486 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.278502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.278628 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.282333 (+ 3705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.282338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.282363 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.282709 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.282720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.284169 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.284174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.286400 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.286419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.288135 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.288142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.288340 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.288375 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.288478 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.288624 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260810 19:20:07.289016 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.278858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.7.43.3:39199)
0810 19:20:07.281984 (+ 3126us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.281992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.282013 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:20:07.282091 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.282097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.282187 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.282296 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.282849 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.284039 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.286604 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.287893 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.288037 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.288166 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.288518 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.288674 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2984,"thread_start_us":2951,"threads_started":1}
I20260810 19:20:07.289845 19557 tablet_service.cc:1467] Processing CreateTablet for tablet 39002146cc134371b042eccb463d49f8 (DEFAULT_TABLE table=test-table [id=ab0d4cee7f88481d988d1a77035411bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:07.290380 19557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39002146cc134371b042eccb463d49f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:07.294523 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.278844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.7.43.1:36603)
0810 19:20:07.288558 (+ 9714us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.288565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.288583 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:20:07.288651 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.288656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.288728 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.289783 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.289924 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.290855 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.292762 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.293622 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.293748 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.293857 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.294198 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.294298 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9589,"thread_start_us":9960,"threads_started":1}
I20260810 19:20:07.294647 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.278419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:36603 (local address 127.0.0.1:59790)
0810 19:20:07.279026 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.279044 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.279153 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.288974 (+ 9821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.288979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.289005 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.289342 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.289354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.291014 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.291019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.292594 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.292607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.293846 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.293852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.294032 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.294066 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.294164 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.294293 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":698,"threads_started":1}
I20260810 19:20:07.295423 19045 tablet_service.cc:1467] Processing CreateTablet for tablet 39002146cc134371b042eccb463d49f8 (DEFAULT_TABLE table=test-table [id=ab0d4cee7f88481d988d1a77035411bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:07.295863 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39002146cc134371b042eccb463d49f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:07.300064 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.277641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35867 (local address 127.0.0.1:41228)
0810 19:20:07.277881 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.277900 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.278042 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.293476 (+ 15434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.293482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.293510 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.293705 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.293717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.296023 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.296028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.297790 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.297807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.299465 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.299471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.299640 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.299670 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.299753 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.299875 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260810 19:20:07.300254 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.277779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.7.43.2:35867)
0810 19:20:07.293159 (+ 15380us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.293165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.293184 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:07.293248 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.293253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.293323 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.293434 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.294884 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.295841 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.298013 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.299287 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.299416 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.299532 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.299835 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.299983 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15265,"thread_start_us":15173,"threads_started":1}
I20260810 19:20:07.300930 19316 tablet_service.cc:1467] Processing CreateTablet for tablet 39002146cc134371b042eccb463d49f8 (DEFAULT_TABLE table=test-table [id=ab0d4cee7f88481d988d1a77035411bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:07.301333 19316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39002146cc134371b042eccb463d49f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:07.302749 19838 tablet_bootstrap.cc:492] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Bootstrap starting.
I20260810 19:20:07.304054 19839 tablet_bootstrap.cc:492] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Bootstrap starting.
I20260810 19:20:07.304505 19838 tablet_bootstrap.cc:654] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:07.308452 19839 tablet_bootstrap.cc:654] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:07.309155 19842 tablet_bootstrap.cc:492] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Bootstrap starting.
I20260810 19:20:07.310077 19838 tablet_bootstrap.cc:492] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: No bootstrap required, opened a new log
I20260810 19:20:07.310214 19838 ts_tablet_manager.cc:1397] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260810 19:20:07.310851 19838 raft_consensus.cc:348] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.311033 19838 raft_consensus.cc:374] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:07.311086 19842 tablet_bootstrap.cc:654] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:07.311127 19838 raft_consensus.cc:729] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64aaeddbe424cea978f4db16dfd3778, State: Initialized, Role: FOLLOWER
I20260810 19:20:07.311368 19838 consensus_queue.cc:260] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.312114 19838 ts_tablet_manager.cc:1428] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:20:07.315968 19178 heartbeater.cc:503] Master 127.7.43.61:41565 was elected leader, sending a full tablet report...
I20260810 19:20:07.318178 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.312711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:33007)
0810 19:20:07.317732 (+ 5021us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.317884 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4877,"thread_start_us":4685,"threads_started":1}
I20260810 19:20:07.320349 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.319940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:55343)
0810 19:20:07.320102 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.320196 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 19:20:07.321780 19842 tablet_bootstrap.cc:492] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: No bootstrap required, opened a new log
I20260810 19:20:07.321856 19839 tablet_bootstrap.cc:492] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: No bootstrap required, opened a new log
I20260810 19:20:07.321913 19842 ts_tablet_manager.cc:1397] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260810 19:20:07.322012 19839 ts_tablet_manager.cc:1397] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20260810 19:20:07.322583 19842 raft_consensus.cc:348] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.322772 19842 raft_consensus.cc:374] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:07.322649 19839 raft_consensus.cc:348] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.322853 19839 raft_consensus.cc:374] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:07.322855 19842 raft_consensus.cc:729] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb44bebcde3747aca0b1d06285d1a392, State: Initialized, Role: FOLLOWER
I20260810 19:20:07.322943 19839 raft_consensus.cc:729] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebf7e6e27244a969ae4a741d2d4093a, State: Initialized, Role: FOLLOWER
I20260810 19:20:07.323066 19842 consensus_queue.cc:260] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.323182 19839 consensus_queue.cc:260] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.323828 19842 ts_tablet_manager.cc:1428] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:07.323939 19839 ts_tablet_manager.cc:1428] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:20:07.328277 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.325100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:49239)
0810 19:20:07.327899 (+ 2799us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.328020 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2665,"thread_start_us":2503,"threads_started":1}
I20260810 19:20:07.329636 19677 heartbeater.cc:503] Master 127.7.43.61:41565 was elected leader, sending a full tablet report...
I20260810 19:20:07.330027 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.328891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:49891)
0810 19:20:07.329741 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.329839 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745,"threads_started":1}
I20260810 19:20:07.330682 19435 heartbeater.cc:503] Master 127.7.43.61:41565 was elected leader, sending a full tablet report...
I20260810 19:20:07.331952 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.324109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:33835)
0810 19:20:07.331535 (+ 7426us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.331689 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7296,"thread_start_us":7130,"threads_started":1}
I20260810 19:20:07.332824 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.332567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:60663)
0810 19:20:07.332670 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.332734 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 19:20:07.602257 19844 raft_consensus.cc:482] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:07.602519 19844 raft_consensus.cc:504] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.604228 19844 leader_election.cc:288] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867), bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199)
I20260810 19:20:07.611455 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.604677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:43211 (local address 127.7.43.3:39199)
0810 19:20:07.604890 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.604896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.605467 (+ 571us) server_negotiation.cc:407] Connection header received
0810 19:20:07.605684 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.605689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.605763 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.605857 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.606395 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.607408 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.609672 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.610454 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.610587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.610694 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.611058 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.611203 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:20:07.611711 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.604679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:50905 (local address 127.7.43.2:35867)
0810 19:20:07.604889 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.604896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.604993 (+ 97us) server_negotiation.cc:407] Connection header received
0810 19:20:07.605258 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.605264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.605345 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.605470 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:07.606114 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.606980 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.609324 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.610702 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.610822 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.610917 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.611319 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.611457 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 19:20:07.611814 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.604592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39199 (local address 127.7.43.1:43211)
0810 19:20:07.605345 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.605362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.605493 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.605878 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.605882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.605908 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.606257 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.606268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.607548 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.607556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.609499 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.609532 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.610647 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.610654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.610827 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.610866 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.611014 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.611149 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":26,"thread_start_us":468,"threads_started":1}
I20260810 19:20:07.612314 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.604538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35867 (local address 127.7.43.1:50905)
0810 19:20:07.604854 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.604877 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:07.605082 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:07.605567 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:07.605572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:07.605602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:07.605932 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.605947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.607102 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.607107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.609146 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:07.609162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.610921 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.610928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:07.611134 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.611164 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 19:20:07.611966 (+ 802us) client_negotiation.cc:240] Negotiation successful
0810 19:20:07.612125 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":18}
I20260810 19:20:07.612589 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39002146cc134371b042eccb463d49f8" candidate_uuid: "c64aaeddbe424cea978f4db16dfd3778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebf7e6e27244a969ae4a741d2d4093a" is_pre_election: true
I20260810 19:20:07.612876 19593 raft_consensus.cc:2454] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64aaeddbe424cea978f4db16dfd3778 in term 0.
I20260810 19:20:07.613175 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39002146cc134371b042eccb463d49f8" candidate_uuid: "c64aaeddbe424cea978f4db16dfd3778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb44bebcde3747aca0b1d06285d1a392" is_pre_election: true
I20260810 19:20:07.613392 18973 leader_election.cc:302] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bebf7e6e27244a969ae4a741d2d4093a, c64aaeddbe424cea978f4db16dfd3778; no voters:
I20260810 19:20:07.613693 19349 raft_consensus.cc:2454] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64aaeddbe424cea978f4db16dfd3778 in term 0.
I20260810 19:20:07.613811 19844 raft_consensus.cc:2781] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:07.613929 19844 raft_consensus.cc:482] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:07.614032 19844 raft_consensus.cc:3037] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:07.616503 19844 raft_consensus.cc:504] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.617195 19844 leader_election.cc:288] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [CANDIDATE]: Term 1 election: Requested vote from peers fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867), bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199)
I20260810 19:20:07.617705 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39002146cc134371b042eccb463d49f8" candidate_uuid: "c64aaeddbe424cea978f4db16dfd3778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb44bebcde3747aca0b1d06285d1a392"
I20260810 19:20:07.617750 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39002146cc134371b042eccb463d49f8" candidate_uuid: "c64aaeddbe424cea978f4db16dfd3778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebf7e6e27244a969ae4a741d2d4093a"
I20260810 19:20:07.617937 19349 raft_consensus.cc:3037] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:07.617957 19593 raft_consensus.cc:3037] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:07.620931 19593 raft_consensus.cc:2454] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64aaeddbe424cea978f4db16dfd3778 in term 1.
I20260810 19:20:07.620931 19349 raft_consensus.cc:2454] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64aaeddbe424cea978f4db16dfd3778 in term 1.
I20260810 19:20:07.621400 18974 leader_election.cc:302] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c64aaeddbe424cea978f4db16dfd3778, fb44bebcde3747aca0b1d06285d1a392; no voters:
I20260810 19:20:07.621801 19844 raft_consensus.cc:2781] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:07.622259 19844 raft_consensus.cc:686] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 1 LEADER]: Becoming Leader. State: Replica: c64aaeddbe424cea978f4db16dfd3778, State: Running, Role: LEADER
I20260810 19:20:07.622912 19844 consensus_queue.cc:237] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } }
I20260810 19:20:07.623407 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.622915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:56041)
0810 19:20:07.623108 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.623238 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 19:20:07.626820 18537 catalog_manager.cc:5495] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 reported cstate change: term changed from 0 to 1, leader changed from <none> to c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "c64aaeddbe424cea978f4db16dfd3778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c64aaeddbe424cea978f4db16dfd3778" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 36603 } health_report { overall_health: HEALTHY } } }
I20260810 19:20:07.656665 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.650730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.7.43.1:36603)
0810 19:20:07.650957 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.650961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.650980 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:07.651151 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.651155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.651256 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.651412 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:07.651865 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.652668 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.654122 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.654616 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.654735 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.655257 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:07.655999 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:07.656132 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.656265 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.656358 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:20:07.661305 19593 raft_consensus.cc:1261] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Refusing update from remote peer c64aaeddbe424cea978f4db16dfd3778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:07.661690 19349 raft_consensus.cc:1261] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Refusing update from remote peer c64aaeddbe424cea978f4db16dfd3778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:07.662149 19864 consensus_queue.cc:1035] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebf7e6e27244a969ae4a741d2d4093a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:07.662626 19844 consensus_queue.cc:1035] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb44bebcde3747aca0b1d06285d1a392" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:07.665539 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.664891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:55523)
0810 19:20:07.665156 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.665285 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 19:20:07.666126 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.665367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:33985)
0810 19:20:07.665669 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.665847 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 19:20:07.671723 19872 mvcc.cc:205] Tried to move back new op lower bound from 7317051832973254656 to 7317051832828211200. Current Snapshot: MvccSnapshot[applied={T|T < 7317051832973254656}]
I20260810 19:20:07.676682 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.674959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.1:45129)
0810 19:20:07.676410 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.676519 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1294}
I20260810 19:20:07.677188 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.675128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.3:48381)
0810 19:20:07.676941 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.677029 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1697}
I20260810 19:20:07.678386 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.677857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.7.43.2:47001)
0810 19:20:07.678140 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.678231 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260810 19:20:07.690959 18537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:07.691264 18537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:07.697671 18537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77cf2ae35eb4c8b986a5f5979d78074: Sending DeleteTablet for 3 replicas of tablet 39002146cc134371b042eccb463d49f8
I20260810 19:20:07.698843 19316 tablet_service.cc:1514] Processing DeleteTablet for tablet 39002146cc134371b042eccb463d49f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:07 UTC) from {username='oryx', principal='oryx/127.7.43.61@KRBTEST.COM'} at 127.0.0.1:41228
I20260810 19:20:07.699579 19557 tablet_service.cc:1514] Processing DeleteTablet for tablet 39002146cc134371b042eccb463d49f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:07 UTC) from {username='oryx', principal='oryx/127.7.43.61@KRBTEST.COM'} at 127.0.0.1:54066
I20260810 19:20:07.699823 19886 tablet_replica.cc:330] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: stopping tablet replica
I20260810 19:20:07.700037 19886 raft_consensus.cc:2229] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:07.700027 19045 tablet_service.cc:1514] Processing DeleteTablet for tablet 39002146cc134371b042eccb463d49f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:07 UTC) from {username='oryx', principal='oryx/127.7.43.61@KRBTEST.COM'} at 127.0.0.1:59790
I20260810 19:20:07.700335 19886 raft_consensus.cc:2258] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:07.702500 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 18821
I20260810 19:20:07.706523 19887 tablet_replica.cc:330] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: stopping tablet replica
I20260810 19:20:07.706550 19886 ts_tablet_manager.cc:1905] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:07.706821 19887 raft_consensus.cc:2229] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:07.707141 19887 raft_consensus.cc:2258] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:07.708427 19887 ts_tablet_manager.cc:1905] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:07.712679 19891 tablet_replica.cc:330] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: stopping tablet replica
I20260810 19:20:07.712882 19891 raft_consensus.cc:2229] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:07.713555 19891 raft_consensus.cc:2258] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:07.714797 19886 ts_tablet_manager.cc:1918] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:07.714906 19891 ts_tablet_manager.cc:1905] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:07.715023 19886 log.cc:1196] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/39002146cc134371b042eccb463d49f8
I20260810 19:20:07.715107 19887 ts_tablet_manager.cc:1918] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:07.715340 19887 log.cc:1196] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/39002146cc134371b042eccb463d49f8
I20260810 19:20:07.715513 19886 ts_tablet_manager.cc:1939] T 39002146cc134371b042eccb463d49f8 P fb44bebcde3747aca0b1d06285d1a392: Deleting consensus metadata
I20260810 19:20:07.715835 19887 ts_tablet_manager.cc:1939] T 39002146cc134371b042eccb463d49f8 P bebf7e6e27244a969ae4a741d2d4093a: Deleting consensus metadata
I20260810 19:20:07.717223 18514 catalog_manager.cc:4841] TS fb44bebcde3747aca0b1d06285d1a392 (127.7.43.2:35867): tablet 39002146cc134371b042eccb463d49f8 (table test-table [id=ab0d4cee7f88481d988d1a77035411bd]) successfully deleted
I20260810 19:20:07.717694 18513 catalog_manager.cc:4841] TS bebf7e6e27244a969ae4a741d2d4093a (127.7.43.3:39199): tablet 39002146cc134371b042eccb463d49f8 (table test-table [id=ab0d4cee7f88481d988d1a77035411bd]) successfully deleted
I20260810 19:20:07.722216 19891 ts_tablet_manager.cc:1918] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:07.722383 19891 log.cc:1196] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/39002146cc134371b042eccb463d49f8
I20260810 19:20:07.722870 19891 ts_tablet_manager.cc:1939] T 39002146cc134371b042eccb463d49f8 P c64aaeddbe424cea978f4db16dfd3778: Deleting consensus metadata
I20260810 19:20:07.724125 18517 catalog_manager.cc:4841] TS c64aaeddbe424cea978f4db16dfd3778 (127.7.43.1:36603): tablet 39002146cc134371b042eccb463d49f8 (table test-table [id=ab0d4cee7f88481d988d1a77035411bd]) successfully deleted
I20260810 19:20:07.730472 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.703278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.7.43.1:36603)
0810 19:20:07.703476 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.703483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.704224 (+ 741us) server_negotiation.cc:407] Connection header received
0810 19:20:07.704389 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.704396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.704494 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.704600 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:07.717148 (+ 12548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.718288 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.720099 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.720768 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.720908 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.724615 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:07.724630 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:07.724648 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:07.724681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:07.727654 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:07.728352 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:07.728359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:07.728362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:07.728508 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:07.728889 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:07.728892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:07.728897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:07.729280 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:07.729619 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.729909 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.730124 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:20:07.731117 19165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:07.832821 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 18821
I20260810 19:20:07.861081 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 19194
I20260810 19:20:07.865576 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.864841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37278)
0810 19:20:07.865293 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.865371 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":247,"threads_started":1}
I20260810 19:20:07.866856 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.866466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:38017 (local address 127.0.0.1:37284)
0810 19:20:07.866625 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:07.866714 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:38017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 19:20:07.874096 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:07.861831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.7.43.2:35867)
0810 19:20:07.862037 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:07.862042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:07.862089 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:20:07.862253 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:07.862257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:07.862320 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:07.862408 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:07.863649 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.864384 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.865842 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:07.866403 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:07.866541 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:07.869116 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:07.869132 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:07.869151 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:07.869185 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:07.871801 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:07.872416 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:07.872422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:07.872424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:07.872559 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:07.872849 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:07.872852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:07.872853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:07.873192 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:07.873421 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:07.873673 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 19:20:07.873850 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:20:07.874737 19426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:07.993325 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 19194
I20260810 19:20:08.022173 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 19454
I20260810 19:20:08.034958 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:08.022993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.7.43.3:39199)
0810 19:20:08.023176 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:08.023181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:08.023200 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:08.023280 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:08.023283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:08.023346 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:08.023426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:08.024865 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:08.025671 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:08.026841 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:08.027226 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:08.027318 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:08.029648 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:08.029665 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:08.029683 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:08.029731 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:08.032491 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:08.033136 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:08.033145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:08.033147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:08.033289 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:08.033600 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:08.033603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:08.033605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:08.033992 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:08.034222 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:08.034427 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 19:20:08.034593 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:20:08.035687 19661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:08.128499 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 19454
I20260810 19:20:08.154229 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 18304
I20260810 19:20:08.155441 18581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:08.278142 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 18304
W20260810 19:20:08.306892 18829 connection.cc:535] server connection from 127.0.0.1:60914 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:20:08.307274 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 18481
I20260810 19:20:08.308281 18894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:08.424019 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 18481
2026-08-10T19:20:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10430 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:20:08 dist-test-slave-7lbf krb5kdc[19910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:20:08 dist-test-slave-7lbf krb5kdc[19910](info): set up 2 sockets
Aug 10 19:20:08 dist-test-slave-7lbf krb5kdc[19910](info): commencing operation
krb5kdc: starting...
W20260810 19:20:09.668807 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:20:09 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab.
Entry for principal oryx/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab.
I20260810 19:20:09.938640 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40505
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab
--principal=oryx/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:10.275111 19932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:10.275414 19932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:10.275547 19932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:10.283079 19932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:10.283185 19932 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 19:20:10.283248 19932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:10.283335 19932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:10.283401 19932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:10.283489 19932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:10.293445 19932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:40505
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.62.keytab
--principal=oryx/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:10.295051 19932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:10.296478 19932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:10.308240 19944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:10.308159 19949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:10.308202 19947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:10.309113 19932 server_base.cc:1034] running on GCE node
I20260810 19:20:10.309952 19932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:10.311379 19932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:10.312613 19932 hybrid_clock.cc:648] HybridClock initialized: now 1786389610312544 us; error 36 us; skew 500 ppm
I20260810 19:20:10.313028 19932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:10 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389610, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:10.316210 19932 init.cc:377] Logged in from keytab as oryx/127.7.43.62@KRBTEST.COM (short username oryx)
I20260810 19:20:10.317698 19932 webserver.cc:460] Webserver started at http://127.7.43.62:41027/ using document root <none> and password file <none>
I20260810 19:20:10.318406 19932 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:10.318514 19932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:10.318847 19932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:10.322034 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1"
format_stamp: "Formatted at 2026-08-10 19:20:10 on dist-test-slave-7lbf"
server_key: "ac1a1582ddcea390cbc6bc524569c6cd"
server_key_iv: "1bb8678f5906b801d8e891f3a04a9c86"
server_key_version: "encryptionkey@0"
I20260810 19:20:10.322840 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1"
format_stamp: "Formatted at 2026-08-10 19:20:10 on dist-test-slave-7lbf"
server_key: "ac1a1582ddcea390cbc6bc524569c6cd"
server_key_iv: "1bb8678f5906b801d8e891f3a04a9c86"
server_key_version: "encryptionkey@0"
I20260810 19:20:10.328090 19932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:20:10.331709 19957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:10.333155 19932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:20:10.333324 19932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1"
format_stamp: "Formatted at 2026-08-10 19:20:10 on dist-test-slave-7lbf"
server_key: "ac1a1582ddcea390cbc6bc524569c6cd"
server_key_iv: "1bb8678f5906b801d8e891f3a04a9c86"
server_key_version: "encryptionkey@0"
I20260810 19:20:10.333547 19932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:10.355110 19932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:10.359115 19932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:10.359367 19932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:10.380365 19932 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:40505
I20260810 19:20:10.380386 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:40505 every 8 connection(s)
I20260810 19:20:10.381836 19932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:20:10.383512 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 19932
I20260810 19:20:10.383747 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:20:10.384584 7340 external_mini_cluster.cc:1428] Setting key 86303fa8f7e489bae1ec96786f43ece7
I20260810 19:20:10.386880 20073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:20:10 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:10.400373 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Bootstrap starting.
I20260810 19:20:10.404098 20073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:10.405544 20073 log.cc:824] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:10.408988 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: No bootstrap required, opened a new log
I20260810 19:20:10.409291 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:10.386728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.7.43.62:40505)
0810 19:20:10.387507 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:10.387524 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:10.387576 (+ 52us) server_negotiation.cc:407] Connection header received
0810 19:20:10.389822 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:10.389852 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:10.390399 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:10.390867 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:10.392226 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:10.394330 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:10.395631 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:10.396659 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:10.396856 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:10.401071 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:10.401082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:10.401101 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:10.401153 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:10.404579 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:10.405466 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:10.405473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:10.405480 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:10.405735 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:10.406292 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:10.406298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:10.406302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:10.407099 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:10.407322 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:10.407766 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 19:20:10.408188 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":340,"threads_started":1}
I20260810 19:20:10.416702 20073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40505 } }
I20260810 19:20:10.417193 20073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:10.417322 20073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f3c6d8b3cc41c1b9634e531f97a6f1, State: Initialized, Role: FOLLOWER
I20260810 19:20:10.417973 20073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40505 } }
I20260810 19:20:10.418208 20073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:20:10.418351 20073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:20:10.418517 20073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:10.420701 20073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40505 } }
I20260810 19:20:10.421237 20073 leader_election.cc:302] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f3c6d8b3cc41c1b9634e531f97a6f1; no voters:
I20260810 19:20:10.421911 20073 leader_election.cc:288] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:20:10.422066 20084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:10.422739 20084 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [term 1 LEADER]: Becoming Leader. State: Replica: c6f3c6d8b3cc41c1b9634e531f97a6f1, State: Running, Role: LEADER
I20260810 19:20:10.423439 20084 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40505 } }
I20260810 19:20:10.424155 20073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:10.426237 20085 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40505 } } }
I20260810 19:20:10.426695 20085 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:10.426294 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f3c6d8b3cc41c1b9634e531f97a6f1. Latest consensus state: current_term: 1 leader_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f3c6d8b3cc41c1b9634e531f97a6f1" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40505 } } }
I20260810 19:20:10.429451 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:10.434060 20100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:10.439033 20100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:10.447031 20100 catalog_manager.cc:1347] Generated new cluster ID: 9c53f22473c443f4a83811a2c8b4390d
I20260810 19:20:10.447166 20100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:10.462420 20100 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:20:10.463729 20100 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:10.473286 20100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Generated new TSK 0
I20260810 19:20:10.474179 20100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab.
Entry for principal oryx/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab.
I20260810 19:20:10.547314 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40505
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:10.906704 20117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:10.906985 20117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:10.907135 20117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:10.914537 20117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:10.914664 20117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:10.914783 20117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:20:10.914911 20117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:10.924877 20117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:40505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:10.926250 20117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:10.927608 20117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:10.941326 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:10.941251 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:10.941285 20128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:10.942093 20117 server_base.cc:1034] running on GCE node
I20260810 19:20:10.942716 20117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:10.943645 20117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:10.944860 20117 hybrid_clock.cc:648] HybridClock initialized: now 1786389610944794 us; error 42 us; skew 500 ppm
I20260810 19:20:10.945199 20117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:10 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389610, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:10.948253 20117 init.cc:377] Logged in from keytab as oryx/127.7.43.1@KRBTEST.COM (short username oryx)
I20260810 19:20:10.949713 20117 webserver.cc:460] Webserver started at http://127.7.43.1:36051/ using document root <none> and password file <none>
I20260810 19:20:10.950394 20117 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:10.950498 20117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:10.950820 20117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:10.953799 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
format_stamp: "Formatted at 2026-08-10 19:20:10 on dist-test-slave-7lbf"
server_key: "f601487269f2ff6b0a1a626bbb23464c"
server_key_iv: "8986daebc181b8eba73b2608fa29d27b"
server_key_version: "encryptionkey@0"
I20260810 19:20:10.954625 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
format_stamp: "Formatted at 2026-08-10 19:20:10 on dist-test-slave-7lbf"
server_key: "f601487269f2ff6b0a1a626bbb23464c"
server_key_iv: "8986daebc181b8eba73b2608fa29d27b"
server_key_version: "encryptionkey@0"
I20260810 19:20:10.959951 20117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:20:10.963533 20139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:10.964852 20117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:20:10.965031 20117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
format_stamp: "Formatted at 2026-08-10 19:20:10 on dist-test-slave-7lbf"
server_key: "f601487269f2ff6b0a1a626bbb23464c"
server_key_iv: "8986daebc181b8eba73b2608fa29d27b"
server_key_version: "encryptionkey@0"
I20260810 19:20:10.965240 20117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:10.979700 20117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:10.983500 20117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:10.983743 20117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:10.986717 20117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:10.986804 20117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:10.986989 20117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:10.987124 20117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:20:10 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389610, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:11.010816 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:10.987624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.7.43.1:53093)
0810 19:20:10.988531 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:10.988587 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:10.990568 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:10.991343 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:10.991359 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:10.992184 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:10.993063 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:10.993106 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:10.994636 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:10.994642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:10.997154 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:10.997187 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:10.998033 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:10.998042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:10.998240 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:10.998966 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:10.999004 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:11.002112 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:11.005572 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.005581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.005604 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.006232 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.006690 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.006695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.006703 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.006994 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.007652 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:11.007661 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:11.008181 (+ 520us) client_negotiation.cc:769] Sending connection context
0810 19:20:11.008532 (+ 351us) client_negotiation.cc:240] Negotiation successful
0810 19:20:11.009150 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":356,"threads_started":1}
I20260810 19:20:11.011745 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:10.987099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:53093 (local address 127.7.43.62:40505)
0810 19:20:10.987598 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:10.987605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:10.988719 (+ 1114us) server_negotiation.cc:407] Connection header received
0810 19:20:10.990852 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:10.990859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:10.990950 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:10.991083 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:10.993454 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:10.994475 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:10.997339 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:10.997789 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:10.997909 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.002427 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:11.002433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:11.002438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:11.002492 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:11.005399 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.006364 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.006369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.006372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.006556 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.007122 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.007128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.007132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.007501 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:11.007662 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:11.011180 (+ 3518us) server_negotiation.cc:299] Negotiation successful
0810 19:20:11.011368 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260810 19:20:11.032999 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:11.018121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:47073 (local address 127.7.43.62:40505)
0810 19:20:11.018314 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:11.018322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:11.018900 (+ 578us) server_negotiation.cc:407] Connection header received
0810 19:20:11.019086 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:11.019090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:11.019154 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:11.019224 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:11.020504 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.021197 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.023263 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.023690 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.023805 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.026367 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:11.026372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:11.026376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:11.026409 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:11.029605 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.030640 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.030647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.030650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.030809 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.031256 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.031262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.031265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.031710 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:11.031884 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:11.032409 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 19:20:11.032637 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:20:11.033129 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:11.018100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.7.43.1:47073)
0810 19:20:11.018782 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:11.018804 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:11.018965 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:11.019275 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:11.019280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:11.019928 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:11.020326 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:11.020340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.021317 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.021322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:11.023100 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:11.023118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.023878 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.023884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:11.024086 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.024847 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:11.024870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:11.026176 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:11.029797 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.029802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.029809 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.030496 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.030932 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.030937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.030940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.031124 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.031850 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:11.031859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:11.032102 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 19:20:11.032280 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 19:20:11.032650 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":345,"threads_started":1}
I20260810 19:20:11.034914 20117 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:38099
I20260810 19:20:11.034965 20336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:38099 every 8 connection(s)
I20260810 19:20:11.036186 20117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:11.038594 20337 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40505
I20260810 19:20:11.038786 20337 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:11.039412 20337 heartbeater.cc:511] Master 127.7.43.62:40505 requested a full tablet report, sending...
I20260810 19:20:11.041286 19983 ts_manager.cc:194] Registered new tserver with Master: 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099)
I20260810 19:20:11.042835 19983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:53093
I20260810 19:20:11.044552 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 20117
I20260810 19:20:11.044749 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:20:11.045408 7340 external_mini_cluster.cc:1428] Setting key dc2b625843d8d5412030484191096c66
WARNING: no policy specified for oryx/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab.
Entry for principal oryx/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab.
I20260810 19:20:11.102766 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40505
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:11.438788 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:11.439086 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:11.439200 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:11.446559 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:11.446686 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:11.446801 20342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:20:11.446910 20342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:11.456756 20342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:40505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:11.458148 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:11.459616 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:11.472910 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:11.472931 20358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:11.472872 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:11.473758 20342 server_base.cc:1034] running on GCE node
I20260810 19:20:11.474414 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:11.475363 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:11.476584 20342 hybrid_clock.cc:648] HybridClock initialized: now 1786389611476504 us; error 50 us; skew 500 ppm
I20260810 19:20:11.476878 20342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:11 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389611, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:11.480029 20342 init.cc:377] Logged in from keytab as oryx/127.7.43.2@KRBTEST.COM (short username oryx)
I20260810 19:20:11.481424 20342 webserver.cc:460] Webserver started at http://127.7.43.2:42753/ using document root <none> and password file <none>
I20260810 19:20:11.482131 20342 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:11.482235 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:11.482555 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:11.485502 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9"
format_stamp: "Formatted at 2026-08-10 19:20:11 on dist-test-slave-7lbf"
server_key: "e08d1ec61c9428ada165d870cfb9c890"
server_key_iv: "d6033df093f223b9a1ee5155b5f47f43"
server_key_version: "encryptionkey@0"
I20260810 19:20:11.486318 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9"
format_stamp: "Formatted at 2026-08-10 19:20:11 on dist-test-slave-7lbf"
server_key: "e08d1ec61c9428ada165d870cfb9c890"
server_key_iv: "d6033df093f223b9a1ee5155b5f47f43"
server_key_version: "encryptionkey@0"
I20260810 19:20:11.491403 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:20:11.495468 20375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:11.497184 20342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:20:11.497407 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9"
format_stamp: "Formatted at 2026-08-10 19:20:11 on dist-test-slave-7lbf"
server_key: "e08d1ec61c9428ada165d870cfb9c890"
server_key_iv: "d6033df093f223b9a1ee5155b5f47f43"
server_key_version: "encryptionkey@0"
I20260810 19:20:11.497731 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:11.515897 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:11.527570 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:11.527850 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:11.531163 20342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:11.531270 20342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:11.531406 20342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:11.531524 20342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 19:20:11 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389611, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:11.558074 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:11.531639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:35963 (local address 127.7.43.62:40505)
0810 19:20:11.531826 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:11.531833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:11.533806 (+ 1973us) server_negotiation.cc:407] Connection header received
0810 19:20:11.537306 (+ 3500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:11.537313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:11.537413 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:11.537556 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:11.540382 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.541610 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.544406 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.545062 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.545164 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.550498 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:11.550505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:11.550511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:11.550567 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:11.553889 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.554931 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.554937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.554940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.555096 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.555682 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.555688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.555692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.556089 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:11.556223 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:11.557400 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0810 19:20:11.557675 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:20:11.560365 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:11.532353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.7.43.2:35963)
0810 19:20:11.533615 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:11.533679 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:11.536835 (+ 3156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:11.538209 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:11.538221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:11.539124 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:11.540159 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:11.540192 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.541805 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.541811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:11.544205 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:11.544222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.545599 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.545611 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:11.545958 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.546842 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:11.546873 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:11.550247 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:11.554052 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.554064 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.554083 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.554788 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.555233 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.555238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.555243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.555540 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.556331 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:11.556345 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:11.556917 (+ 572us) client_negotiation.cc:769] Sending connection context
0810 19:20:11.558383 (+ 1466us) client_negotiation.cc:240] Negotiation successful
0810 19:20:11.558916 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":399,"threads_started":1}
I20260810 19:20:11.582409 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:11.566487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:39137 (local address 127.7.43.62:40505)
0810 19:20:11.566653 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:11.566659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:11.567300 (+ 641us) server_negotiation.cc:407] Connection header received
0810 19:20:11.567535 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:11.567540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:11.567614 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:11.567676 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:11.569179 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.570348 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.572433 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.573059 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.573154 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.575734 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:11.575741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:11.575746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:11.575784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:11.579135 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.580031 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.580037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.580040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.580188 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:11.580631 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:11.580636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:11.580639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:11.581064 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:11.581182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:11.581827 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 19:20:11.582066 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:20:11.583491 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:11.566491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.7.43.2:39137)
0810 19:20:11.567204 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:11.567223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:11.567403 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:11.567874 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:11.567880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:11.568526 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:11.568993 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:11.569008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.570594 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.570599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:11.572268 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:11.572285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:11.573438 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:11.573444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:11.573651 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:11.574347 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:11.574373 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:11.575525 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:11.579291 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.579297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.579300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.579875 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.580341 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:11.580346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:11.580350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:11.580505 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:11.581285 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:11.581290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:11.581534 (+ 244us) client_negotiation.cc:769] Sending connection context
0810 19:20:11.582623 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0810 19:20:11.582984 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":397,"threads_started":1}
I20260810 19:20:11.592638 20342 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45027
I20260810 19:20:11.592661 20575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45027 every 8 connection(s)
I20260810 19:20:11.593869 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:11.595888 20576 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40505
I20260810 19:20:11.596105 20576 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:11.596709 20576 heartbeater.cc:511] Master 127.7.43.62:40505 requested a full tablet report, sending...
I20260810 19:20:11.598112 19983 ts_manager.cc:194] Registered new tserver with Master: 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027)
I20260810 19:20:11.599215 19983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:35963
I20260810 19:20:11.601963 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 20342
I20260810 19:20:11.602140 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:20:11.602630 7340 external_mini_cluster.cc:1428] Setting key caa734ec36be02878b4ff25ae593e2ba
WARNING: no policy specified for oryx/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab.
Entry for principal oryx/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab.
I20260810 19:20:11.667052 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:40505
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:11.995707 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:11.995993 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:11.996126 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:12.003386 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:12.003499 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:12.003624 20585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 19:20:12.003731 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:12.013437 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.43.62:40505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:12.014809 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:12.016177 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:12.028978 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:12.028934 20600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:12.029444 20593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:12.029987 20585 server_base.cc:1034] running on GCE node
I20260810 19:20:12.030617 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:12.031575 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:12.032791 20585 hybrid_clock.cc:648] HybridClock initialized: now 1786389612032713 us; error 55 us; skew 500 ppm
I20260810 19:20:12.033092 20585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389612, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:12.036854 20585 init.cc:377] Logged in from keytab as oryx/127.7.43.3@KRBTEST.COM (short username oryx)
I20260810 19:20:12.040791 20585 webserver.cc:460] Webserver started at http://127.7.43.3:36145/ using document root <none> and password file <none>
I20260810 19:20:12.041508 20585 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:12.041657 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:12.041999 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:12.044947 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "11574d84791a4ff8b2286314bad5dc40"
format_stamp: "Formatted at 2026-08-10 19:20:12 on dist-test-slave-7lbf"
server_key: "6bb575761bf27f35f8157c8b21275efe"
server_key_iv: "01869810dacf6f9e8755cf97d6cffc38"
server_key_version: "encryptionkey@0"
I20260810 19:20:12.045573 20337 heartbeater.cc:503] Master 127.7.43.62:40505 was elected leader, sending a full tablet report...
I20260810 19:20:12.045852 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "11574d84791a4ff8b2286314bad5dc40"
format_stamp: "Formatted at 2026-08-10 19:20:12 on dist-test-slave-7lbf"
server_key: "6bb575761bf27f35f8157c8b21275efe"
server_key_iv: "01869810dacf6f9e8755cf97d6cffc38"
server_key_version: "encryptionkey@0"
I20260810 19:20:12.051385 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:20:12.055001 20615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:12.056303 20585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 19:20:12.056465 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "11574d84791a4ff8b2286314bad5dc40"
format_stamp: "Formatted at 2026-08-10 19:20:12 on dist-test-slave-7lbf"
server_key: "6bb575761bf27f35f8157c8b21275efe"
server_key_iv: "01869810dacf6f9e8755cf97d6cffc38"
server_key_version: "encryptionkey@0"
I20260810 19:20:12.056699 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:12.074800 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:12.079094 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:12.079344 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:12.082104 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:12.082206 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:12.082341 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:12.082420 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389612, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:12.106403 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.082664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:60885 (local address 127.7.43.62:40505)
0810 19:20:12.083627 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.083631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.084649 (+ 1018us) server_negotiation.cc:407] Connection header received
0810 19:20:12.086752 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.086757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.086844 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.086941 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:12.089117 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.090319 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.092822 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.093481 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.093641 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.099104 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:12.099111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:12.099115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:12.099156 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:12.102551 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.103504 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.103509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.103512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.103663 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.104244 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.104250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.104254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.104575 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:12.104703 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.105746 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.105975 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":848,"thread_start_us":714,"threads_started":1}
I20260810 19:20:12.107898 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.083483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.7.43.3:60885)
0810 19:20:12.084484 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.084520 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.086480 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.087206 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.087216 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.087936 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:12.088842 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.088863 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.090525 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.090545 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.092681 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.092698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.093983 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.093994 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.094321 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.095182 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:12.095218 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:12.098822 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:12.102705 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:12.102718 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:12.102731 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:12.103376 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:12.103789 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:12.103794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:12.103798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:12.104122 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:12.104726 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:12.104734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:12.105240 (+ 506us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.105657 (+ 417us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.106274 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":538,"threads_started":1}
I20260810 19:20:12.122666 20585 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:38239
I20260810 19:20:12.122709 20816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:38239 every 8 connection(s)
I20260810 19:20:12.124030 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:12.126350 20817 heartbeater.cc:347] Connected to a master server at 127.7.43.62:40505
I20260810 19:20:12.126573 20817 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:12.127375 20817 heartbeater.cc:511] Master 127.7.43.62:40505 requested a full tablet report, sending...
I20260810 19:20:12.128161 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.115273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:44861 (local address 127.7.43.62:40505)
0810 19:20:12.115491 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.115498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.115521 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:20:12.115677 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.115683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.115761 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.115869 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:12.116738 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.117863 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.119175 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.119845 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.119976 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.122673 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:12.122681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:12.122686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:12.122728 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:12.125614 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.126349 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.126355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.126357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.126510 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.126902 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.126906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.126908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.127186 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:12.127306 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.127747 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.127916 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 19:20:12.128420 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.114765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.7.43.3:44861)
0810 19:20:12.115433 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.115453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.115572 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.115922 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.115930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.116326 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:12.116613 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.116621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.118021 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.118026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.119048 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.119060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.120246 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.120254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.120501 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.121154 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:12.121178 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:12.122444 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:12.125733 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:12.125739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:12.125743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:12.126215 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:12.126625 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:12.126630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:12.126633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:12.126801 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:12.127307 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:12.127311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:12.127515 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.127669 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.128000 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":383,"threads_started":1}
I20260810 19:20:12.128880 19983 ts_manager.cc:194] Registered new tserver with Master: 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:12.129920 19983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:60885
I20260810 19:20:12.132587 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 20585
I20260810 19:20:12.132772 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:20:12.133183 7340 external_mini_cluster.cc:1428] Setting key 419f5f5c31d8551fd23f56a10b0d74d4
I20260810 19:20:12.136070 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:20:12.194573 20633 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:12.184633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.7.43.62:40505)
0810 19:20:12.184839 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.184846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.185036 (+ 190us) server_negotiation.cc:407] Connection header received
0810 19:20:12.185227 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.185232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.185298 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.185401 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:12.186741 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.187469 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.188625 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.188986 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.189076 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.194305 (+ 5229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38788: BlockingRecv error: recv got EOF from 127.0.0.1:38788 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:12.209509 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.197403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.7.43.62:40505)
0810 19:20:12.197596 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.197600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.197886 (+ 286us) server_negotiation.cc:407] Connection header received
0810 19:20:12.198031 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.198034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.198094 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.198174 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:12.199207 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.199891 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.201033 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.201494 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.201640 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.205039 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:12.205045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:12.205048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:12.205080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:12.207552 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.208102 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.208105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.208108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.208211 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.208476 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.208479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.208481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.208738 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:12.208955 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.209113 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.209274 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:20:12.213474 19983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:12.220314 19983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:12.254438 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.232530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.7.43.2:45027)
0810 19:20:12.246308 (+ 13778us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.246318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.246351 (+ 33us) server_negotiation.cc:407] Connection header received
0810 19:20:12.246460 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.246468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.246749 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.246967 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.248005 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.249310 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.252122 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.253342 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.253481 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.253736 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.253984 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.254152 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13602,"thread_start_us":13477,"threads_started":1}
I20260810 19:20:12.254565 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.232374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45027 (local address 127.0.0.1:47670)
0810 19:20:12.243969 (+ 11595us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.244028 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.244217 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.247216 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.247224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.247265 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.247791 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.247805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.249418 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.249422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.251947 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.251966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.253559 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.253566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.253744 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.253795 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.253933 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.254122 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11320,"mutex_wait_us":64,"thread_start_us":15034,"threads_started":1}
I20260810 19:20:12.255720 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.233497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.7.43.1:38099)
0810 19:20:12.244298 (+ 10801us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.244309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.248105 (+ 3796us) server_negotiation.cc:407] Connection header received
0810 19:20:12.248208 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.248215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.248523 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.248695 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.249414 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.250954 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.253142 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.254511 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.254631 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.254812 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.255273 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.255443 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10640,"thread_start_us":10487,"threads_started":1}
I20260810 19:20:12.256268 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.232518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:38239 (local address 127.0.0.1:48068)
0810 19:20:12.247671 (+ 15153us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.247698 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.247838 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.248505 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.248509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.248535 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.248938 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.248951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.250449 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.250455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.252581 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.252596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.254540 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.254551 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.254716 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.254746 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.254830 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.255843 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15023,"mutex_wait_us":49,"thread_start_us":11038,"threads_started":1}
I20260810 19:20:12.256837 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.233033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.7.43.3:38239)
0810 19:20:12.246827 (+ 13794us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.246836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.247972 (+ 1136us) server_negotiation.cc:407] Connection header received
0810 19:20:12.248070 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.248077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.248313 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.248500 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.249159 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.250317 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.252745 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.254075 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.254186 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.254394 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.256476 (+ 2082us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.256612 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13616,"thread_start_us":13409,"threads_started":1}
I20260810 19:20:12.258452 20462 tablet_service.cc:1467] Processing CreateTablet for tablet 69fad4a8f1b6434bb0a14866876dafcf (DEFAULT_TABLE table=test-table [id=f70b63d0eb1643518f185c3e5f11ce7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:12.259024 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.232874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38099 (local address 127.0.0.1:57648)
0810 19:20:12.247857 (+ 14983us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.247877 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.247983 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.248864 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.248872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.248899 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.249257 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.249265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.251148 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.251153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.252981 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.252995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.254971 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.254978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.255095 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.255123 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.255213 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.255331 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14862,"mutex_wait_us":1,"thread_start_us":14151,"threads_started":1}
I20260810 19:20:12.259771 20462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fad4a8f1b6434bb0a14866876dafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.260124 20702 tablet_service.cc:1467] Processing CreateTablet for tablet 69fad4a8f1b6434bb0a14866876dafcf (DEFAULT_TABLE table=test-table [id=f70b63d0eb1643518f185c3e5f11ce7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:12.261650 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fad4a8f1b6434bb0a14866876dafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.262671 20225 tablet_service.cc:1467] Processing CreateTablet for tablet 69fad4a8f1b6434bb0a14866876dafcf (DEFAULT_TABLE table=test-table [id=f70b63d0eb1643518f185c3e5f11ce7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:12.264020 20225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69fad4a8f1b6434bb0a14866876dafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.271751 20858 tablet_bootstrap.cc:492] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Bootstrap starting.
I20260810 19:20:12.275290 20858 tablet_bootstrap.cc:654] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.275964 20864 tablet_bootstrap.cc:492] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Bootstrap starting.
I20260810 19:20:12.276738 20858 log.cc:824] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:12.279438 20864 tablet_bootstrap.cc:654] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.279654 20869 tablet_bootstrap.cc:492] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Bootstrap starting.
I20260810 19:20:12.279788 20858 tablet_bootstrap.cc:492] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: No bootstrap required, opened a new log
I20260810 19:20:12.280057 20858 ts_tablet_manager.cc:1397] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:20:12.280642 20864 log.cc:824] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:12.282580 20869 tablet_bootstrap.cc:654] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.283615 20869 log.cc:824] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:12.283728 20864 tablet_bootstrap.cc:492] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: No bootstrap required, opened a new log
I20260810 19:20:12.283979 20864 ts_tablet_manager.cc:1397] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:20:12.285966 20869 tablet_bootstrap.cc:492] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: No bootstrap required, opened a new log
I20260810 19:20:12.286192 20869 ts_tablet_manager.cc:1397] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:20:12.290357 20858 raft_consensus.cc:348] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.290817 20858 raft_consensus.cc:374] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.290917 20858 raft_consensus.cc:729] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.291530 20858 consensus_queue.cc:260] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.292644 20576 heartbeater.cc:503] Master 127.7.43.62:40505 was elected leader, sending a full tablet report...
I20260810 19:20:12.293265 20858 ts_tablet_manager.cc:1428] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260810 19:20:12.293571 20869 raft_consensus.cc:348] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.293987 20869 raft_consensus.cc:374] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.294080 20869 raft_consensus.cc:729] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee45652e5a84b43aa8f6c7fdc16a15c, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.293859 20864 raft_consensus.cc:348] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.294258 20864 raft_consensus.cc:374] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.294348 20864 raft_consensus.cc:729] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11574d84791a4ff8b2286314bad5dc40, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.294661 20869 consensus_queue.cc:260] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.295047 20864 consensus_queue.cc:260] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.297497 20869 ts_tablet_manager.cc:1428] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260810 19:20:12.298736 20817 heartbeater.cc:503] Master 127.7.43.62:40505 was elected leader, sending a full tablet report...
I20260810 19:20:12.299249 20864 ts_tablet_manager.cc:1428] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
W20260810 19:20:12.345880 20579 tablet.cc:2369] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:20:12.375765 20819 tablet.cc:2369] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:12.452318 20874 raft_consensus.cc:482] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:12.452641 20874 raft_consensus.cc:504] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.454445 20874 leader_election.cc:288] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239), 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099)
I20260810 19:20:12.461345 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.454953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:37699 (local address 127.7.43.3:38239)
0810 19:20:12.455200 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.455208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.455958 (+ 750us) server_negotiation.cc:407] Connection header received
0810 19:20:12.456111 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.456116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.456192 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.456262 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.456923 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.457712 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.459581 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.460378 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.460474 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.460556 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.460963 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.461091 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 19:20:12.461452 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.455047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:42411 (local address 127.7.43.1:38099)
0810 19:20:12.455251 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.455258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.456071 (+ 813us) server_negotiation.cc:407] Connection header received
0810 19:20:12.456223 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.456227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.456314 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.456404 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.457005 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.457742 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.459586 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.460514 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.460645 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.460746 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.461098 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.461226 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:20:12.461551 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.454853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:38239 (local address 127.7.43.2:37699)
0810 19:20:12.455835 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.455850 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.455975 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.456306 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.456310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.456342 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.456769 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.456782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.457832 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.457837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.459420 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.459432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.460587 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.460595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.460805 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.460829 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.460900 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.461006 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":46,"thread_start_us":552,"threads_started":1}
I20260810 19:20:12.461956 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.454922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38099 (local address 127.7.43.2:42411)
0810 19:20:12.455946 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.455963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.456097 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.456438 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.456441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.456473 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.456819 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.456829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.457877 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.457881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.459434 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.459447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.460720 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.460724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.460863 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.460891 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.461056 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.461175 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":21,"thread_start_us":533,"threads_started":1}
I20260810 19:20:12.462579 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fad4a8f1b6434bb0a14866876dafcf" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40" is_pre_election: true
I20260810 19:20:12.462882 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fad4a8f1b6434bb0a14866876dafcf" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" is_pre_election: true
I20260810 19:20:12.463066 20741 raft_consensus.cc:2454] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 0.
I20260810 19:20:12.463428 20257 raft_consensus.cc:2454] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 0.
I20260810 19:20:12.463675 20376 leader_election.cc:302] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:12.464046 20874 raft_consensus.cc:2781] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:12.464205 20874 raft_consensus.cc:482] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:12.464311 20874 raft_consensus.cc:3037] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.466552 20874 raft_consensus.cc:504] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.467113 20874 leader_election.cc:288] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239), 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099)
I20260810 19:20:12.467504 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fad4a8f1b6434bb0a14866876dafcf" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40"
I20260810 19:20:12.467549 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69fad4a8f1b6434bb0a14866876dafcf" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
I20260810 19:20:12.467675 20741 raft_consensus.cc:3037] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.467711 20257 raft_consensus.cc:3037] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.470036 20741 raft_consensus.cc:2454] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 1.
I20260810 19:20:12.470131 20257 raft_consensus.cc:2454] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 1.
I20260810 19:20:12.470471 20376 leader_election.cc:302] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:12.470762 20874 raft_consensus.cc:2781] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:12.471160 20874 raft_consensus.cc:686] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Running, Role: LEADER
I20260810 19:20:12.471594 20874 consensus_queue.cc:237] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } }
I20260810 19:20:12.475854 19983 catalog_manager.cc:5495] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } health_report { overall_health: UNKNOWN } } }
I20260810 19:20:12.494968 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.489715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.7.43.2:45027)
0810 19:20:12.489937 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.489942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.490068 (+ 126us) server_negotiation.cc:407] Connection header received
0810 19:20:12.490278 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.490282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.490359 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.490481 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:12.490851 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.491575 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.492916 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.493336 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.493424 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.493877 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:12.494429 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:12.494546 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.494611 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.494696 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:20:12.504045 20741 raft_consensus.cc:1261] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Refusing update from remote peer 7da38ef950dc4a36a1b3bb81e9c08ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:12.504151 20257 raft_consensus.cc:1261] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Refusing update from remote peer 7da38ef950dc4a36a1b3bb81e9c08ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:12.505012 20874 consensus_queue.cc:1035] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:12.505321 20881 consensus_queue.cc:1035] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:12.519536 20887 mvcc.cc:205] Tried to move back new op lower bound from 7317051852804952064 to 7317051852687294464. Current Snapshot: MvccSnapshot[applied={T|T < 7317051852804952064}]
I20260810 19:20:12.533972 19980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:12.534250 19980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:12.537915 19980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Sending DeleteTablet for 3 replicas of tablet 69fad4a8f1b6434bb0a14866876dafcf
I20260810 19:20:12.539119 20462 tablet_service.cc:1514] Processing DeleteTablet for tablet 69fad4a8f1b6434bb0a14866876dafcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:12 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260810 19:20:12.539294 20225 tablet_service.cc:1514] Processing DeleteTablet for tablet 69fad4a8f1b6434bb0a14866876dafcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:12 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:57648
I20260810 19:20:12.539315 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet 69fad4a8f1b6434bb0a14866876dafcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:12 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:48068
I20260810 19:20:12.542713 19983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:12.543468 19983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:12.551281 20911 tablet_replica.cc:330] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: stopping tablet replica
I20260810 19:20:12.551988 20911 raft_consensus.cc:2229] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:12.552289 20455 tablet_service.cc:1467] Processing CreateTablet for tablet 9638021f693646bc87f9d3077dc875e3 (DEFAULT_TABLE table=test-table [id=13e434bbbc3c409383639ed0c9e601d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:12.552608 20911 raft_consensus.cc:2258] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:12.553007 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9638021f693646bc87f9d3077dc875e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.555181 20911 ts_tablet_manager.cc:1905] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:12.558185 20223 tablet_service.cc:1467] Processing CreateTablet for tablet 9638021f693646bc87f9d3077dc875e3 (DEFAULT_TABLE table=test-table [id=13e434bbbc3c409383639ed0c9e601d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:12.558404 20904 tablet_replica.cc:330] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: stopping tablet replica
I20260810 19:20:12.558869 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9638021f693646bc87f9d3077dc875e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.558866 20904 raft_consensus.cc:2229] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:12.559564 20904 raft_consensus.cc:2258] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:12.560762 20907 tablet_replica.cc:330] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: stopping tablet replica
I20260810 19:20:12.561232 20907 raft_consensus.cc:2229] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:12.561822 20907 raft_consensus.cc:2258] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:12.562402 20702 tablet_service.cc:1467] Processing CreateTablet for tablet 9638021f693646bc87f9d3077dc875e3 (DEFAULT_TABLE table=test-table [id=13e434bbbc3c409383639ed0c9e601d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:12.562916 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9638021f693646bc87f9d3077dc875e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.562958 20904 ts_tablet_manager.cc:1905] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:12.564042 20907 ts_tablet_manager.cc:1905] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:12.569612 20864 tablet_bootstrap.cc:492] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Bootstrap starting.
I20260810 19:20:12.569578 20904 ts_tablet_manager.cc:1918] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:12.569752 20904 log.cc:1196] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/69fad4a8f1b6434bb0a14866876dafcf
I20260810 19:20:12.570417 20904 ts_tablet_manager.cc:1939] T 69fad4a8f1b6434bb0a14866876dafcf P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting consensus metadata
I20260810 19:20:12.571030 20907 ts_tablet_manager.cc:1918] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:12.571182 20907 log.cc:1196] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/69fad4a8f1b6434bb0a14866876dafcf
I20260810 19:20:12.571316 20864 tablet_bootstrap.cc:654] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.571698 20907 ts_tablet_manager.cc:1939] T 69fad4a8f1b6434bb0a14866876dafcf P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting consensus metadata
I20260810 19:20:12.572480 19961 catalog_manager.cc:4841] TS 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027): tablet 69fad4a8f1b6434bb0a14866876dafcf (table test-table [id=f70b63d0eb1643518f185c3e5f11ce7e]) successfully deleted
I20260810 19:20:12.573432 19959 catalog_manager.cc:4841] TS 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099): tablet 69fad4a8f1b6434bb0a14866876dafcf (table test-table [id=f70b63d0eb1643518f185c3e5f11ce7e]) successfully deleted
I20260810 19:20:12.573956 20864 tablet_bootstrap.cc:492] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: No bootstrap required, opened a new log
I20260810 19:20:12.574113 20864 ts_tablet_manager.cc:1397] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:20:12.574489 20858 tablet_bootstrap.cc:492] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Bootstrap starting.
I20260810 19:20:12.574945 20864 raft_consensus.cc:348] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.575172 20864 raft_consensus.cc:374] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.575254 20864 raft_consensus.cc:729] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11574d84791a4ff8b2286314bad5dc40, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.575539 20864 consensus_queue.cc:260] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.576046 20858 tablet_bootstrap.cc:654] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.576057 20864 ts_tablet_manager.cc:1428] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:12.576635 20869 tablet_bootstrap.cc:492] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Bootstrap starting.
I20260810 19:20:12.576633 20911 ts_tablet_manager.cc:1918] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:12.576774 20911 log.cc:1196] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/69fad4a8f1b6434bb0a14866876dafcf
I20260810 19:20:12.577286 20911 ts_tablet_manager.cc:1939] T 69fad4a8f1b6434bb0a14866876dafcf P 11574d84791a4ff8b2286314bad5dc40: Deleting consensus metadata
I20260810 19:20:12.578219 20869 tablet_bootstrap.cc:654] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.579370 19958 catalog_manager.cc:4841] TS 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239): tablet 69fad4a8f1b6434bb0a14866876dafcf (table test-table [id=f70b63d0eb1643518f185c3e5f11ce7e]) successfully deleted
I20260810 19:20:12.580088 20869 tablet_bootstrap.cc:492] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: No bootstrap required, opened a new log
I20260810 19:20:12.580216 20869 ts_tablet_manager.cc:1397] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260810 19:20:12.580819 20869 raft_consensus.cc:348] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.580984 20869 raft_consensus.cc:374] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.581073 20869 raft_consensus.cc:729] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee45652e5a84b43aa8f6c7fdc16a15c, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.581321 20869 consensus_queue.cc:260] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.581849 20869 ts_tablet_manager.cc:1428] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:12.583019 20858 tablet_bootstrap.cc:492] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: No bootstrap required, opened a new log
I20260810 19:20:12.583218 20858 ts_tablet_manager.cc:1397] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260810 19:20:12.584091 20858 raft_consensus.cc:348] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.584322 20858 raft_consensus.cc:374] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.584434 20858 raft_consensus.cc:729] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.584683 20858 consensus_queue.cc:260] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.585876 20858 ts_tablet_manager.cc:1428] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 19:20:12.655762 20876 raft_consensus.cc:482] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:12.656042 20876 raft_consensus.cc:504] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.657598 20876 leader_election.cc:288] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099), 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027)
I20260810 19:20:12.665768 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.658410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:50121 (local address 127.7.43.2:45027)
0810 19:20:12.658589 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.658595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.659173 (+ 578us) server_negotiation.cc:407] Connection header received
0810 19:20:12.659378 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.659382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.659464 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.659545 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.660283 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.661370 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.663348 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.664700 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.664851 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.664945 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.665300 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.665447 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 19:20:12.665809 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.658101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:55659 (local address 127.7.43.1:38099)
0810 19:20:12.658255 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.658261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.658943 (+ 682us) server_negotiation.cc:407] Connection header received
0810 19:20:12.659257 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.659263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.659371 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.659473 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:12.659935 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.661074 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.663568 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.664755 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.664856 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.664960 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.665425 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.665579 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 19:20:12.665800 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.657930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45027 (local address 127.7.43.3:50121)
0810 19:20:12.659094 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.659109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.659254 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.659661 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.659666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.659698 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.660064 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.660076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.661530 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.661535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.663199 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.663212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.664897 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.664904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.665106 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.665152 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.665253 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.665403 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":2,"thread_start_us":896,"threads_started":1}
I20260810 19:20:12.666666 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.657800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38099 (local address 127.7.43.3:55659)
0810 19:20:12.658806 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.658827 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.659075 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.659497 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.659502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.659526 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:12.659765 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.659775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.661208 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.661214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.663410 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.663426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.665102 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.665107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.665251 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.665282 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.666218 (+ 936us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.666344 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":30,"thread_start_us":780,"threads_started":1}
I20260810 19:20:12.666646 20504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9638021f693646bc87f9d3077dc875e3" candidate_uuid: "11574d84791a4ff8b2286314bad5dc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" is_pre_election: true
I20260810 19:20:12.667157 20504 raft_consensus.cc:2454] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11574d84791a4ff8b2286314bad5dc40 in term 0.
I20260810 19:20:12.667372 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9638021f693646bc87f9d3077dc875e3" candidate_uuid: "11574d84791a4ff8b2286314bad5dc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" is_pre_election: true
I20260810 19:20:12.667625 20257 raft_consensus.cc:2454] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11574d84791a4ff8b2286314bad5dc40 in term 0.
I20260810 19:20:12.667728 20623 leader_election.cc:302] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:12.668130 20876 raft_consensus.cc:2781] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:12.668258 20876 raft_consensus.cc:482] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:12.668356 20876 raft_consensus.cc:3037] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.670980 20876 raft_consensus.cc:504] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.671600 20876 leader_election.cc:288] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099), 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027)
I20260810 19:20:12.671993 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9638021f693646bc87f9d3077dc875e3" candidate_uuid: "11574d84791a4ff8b2286314bad5dc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
I20260810 19:20:12.672209 20257 raft_consensus.cc:3037] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.672358 20504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9638021f693646bc87f9d3077dc875e3" candidate_uuid: "11574d84791a4ff8b2286314bad5dc40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9"
I20260810 19:20:12.672571 20504 raft_consensus.cc:3037] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.674289 20257 raft_consensus.cc:2454] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11574d84791a4ff8b2286314bad5dc40 in term 1.
I20260810 19:20:12.674640 20619 leader_election.cc:302] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 3ee45652e5a84b43aa8f6c7fdc16a15c; no voters:
I20260810 19:20:12.674917 20876 raft_consensus.cc:2781] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:12.675347 20876 raft_consensus.cc:686] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 1 LEADER]: Becoming Leader. State: Replica: 11574d84791a4ff8b2286314bad5dc40, State: Running, Role: LEADER
I20260810 19:20:12.675747 20876 consensus_queue.cc:237] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.676178 20504 raft_consensus.cc:2454] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11574d84791a4ff8b2286314bad5dc40 in term 1.
I20260810 19:20:12.679382 19980 catalog_manager.cc:5495] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "11574d84791a4ff8b2286314bad5dc40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } health_report { overall_health: HEALTHY } } }
Aug 10 19:20:12 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389610, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.62@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:12.741201 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.723779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:40505 (local address 127.0.0.1:38806)
0810 19:20:12.724729 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:12.724747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:12.724904 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:12.725259 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:12.725264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:12.726065 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:12.726304 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.726314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.727386 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.727392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.728315 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:12.728326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.729023 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.729029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:12.729216 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.732951 (+ 3735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:12.732970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:12.735558 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:12.738726 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:12.738739 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:12.738748 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:12.739198 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:12.739533 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:12.739538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:12.739542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:12.739674 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:12.740174 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:12.740183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:12.740344 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 19:20:12.740499 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 19:20:12.740767 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":667,"threads_started":1}
I20260810 19:20:12.741279 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:12.723810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.7.43.62:40505)
0810 19:20:12.724654 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:12.724660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:12.724833 (+ 173us) server_negotiation.cc:407] Connection header received
0810 19:20:12.725018 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:12.725022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:12.725089 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:12.725198 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:12.726475 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.727256 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.728451 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:12.728842 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:12.728956 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:12.735805 (+ 6849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:12.735810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:12.735814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:12.735845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:12.738608 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.739311 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.739316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.739318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.739434 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:12.739770 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:12.739775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:12.739778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:12.740051 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:12.740179 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:12.740602 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 19:20:12.740931 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":598,"thread_start_us":328,"threads_started":1}
I20260810 19:20:12.746994 19980 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:38806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 19:20:12.747977 19980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:12.754753 20223 tablet_service.cc:1467] Processing CreateTablet for tablet f05e8456105e4459a4ea5695958fa95d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f94ba3fb0ed442f84bf3b9445353fe0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:20:12.755296 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05e8456105e4459a4ea5695958fa95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.755618 20455 tablet_service.cc:1467] Processing CreateTablet for tablet f05e8456105e4459a4ea5695958fa95d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f94ba3fb0ed442f84bf3b9445353fe0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:20:12.756160 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05e8456105e4459a4ea5695958fa95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.760476 20702 tablet_service.cc:1467] Processing CreateTablet for tablet f05e8456105e4459a4ea5695958fa95d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f94ba3fb0ed442f84bf3b9445353fe0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 19:20:12.761030 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05e8456105e4459a4ea5695958fa95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:12.761274 20869 tablet_bootstrap.cc:492] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Bootstrap starting.
I20260810 19:20:12.762717 20869 tablet_bootstrap.cc:654] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.764801 20869 tablet_bootstrap.cc:492] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: No bootstrap required, opened a new log
I20260810 19:20:12.764951 20869 ts_tablet_manager.cc:1397] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 19:20:12.765558 20869 raft_consensus.cc:348] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.765767 20869 raft_consensus.cc:374] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.765846 20869 raft_consensus.cc:729] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee45652e5a84b43aa8f6c7fdc16a15c, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.766076 20869 consensus_queue.cc:260] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.766489 20875 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:12.766841 20875 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:12.767272 20869 ts_tablet_manager.cc:1428] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:12.770671 20858 tablet_bootstrap.cc:492] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Bootstrap starting.
I20260810 19:20:12.771682 20864 tablet_bootstrap.cc:492] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: Bootstrap starting.
I20260810 19:20:12.772161 20858 tablet_bootstrap.cc:654] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.773164 20864 tablet_bootstrap.cc:654] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:12.774269 20858 tablet_bootstrap.cc:492] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: No bootstrap required, opened a new log
I20260810 19:20:12.774439 20858 ts_tablet_manager.cc:1397] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260810 19:20:12.775089 20858 raft_consensus.cc:348] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.775276 20864 tablet_bootstrap.cc:492] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: No bootstrap required, opened a new log
I20260810 19:20:12.775280 20858 raft_consensus.cc:374] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.775357 20858 raft_consensus.cc:729] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.775462 20864 ts_tablet_manager.cc:1397] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 19:20:12.775615 20858 consensus_queue.cc:260] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.776317 20858 ts_tablet_manager.cc:1428] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 19:20:12.776062 20898 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:12.776467 20898 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:12.777680 20864 raft_consensus.cc:348] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.777906 20864 raft_consensus.cc:374] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:12.777988 20864 raft_consensus.cc:729] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11574d84791a4ff8b2286314bad5dc40, State: Initialized, Role: FOLLOWER
I20260810 19:20:12.778213 20864 consensus_queue.cc:260] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.778620 20876 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:12.778858 20876 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:12.779512 20864 ts_tablet_manager.cc:1428] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260810 19:20:12.791186 20338 tablet.cc:2369] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:12.946450 20898 raft_consensus.cc:482] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:12.946719 20898 raft_consensus.cc:504] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.947541 20898 leader_election.cc:288] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099), 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:12.948194 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05e8456105e4459a4ea5695958fa95d" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" is_pre_election: true
I20260810 19:20:12.948192 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05e8456105e4459a4ea5695958fa95d" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40" is_pre_election: true
I20260810 19:20:12.948485 20257 raft_consensus.cc:2454] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 0.
I20260810 19:20:12.948520 20741 raft_consensus.cc:2454] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 0.
I20260810 19:20:12.949054 20376 leader_election.cc:302] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:12.949355 20898 raft_consensus.cc:2781] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:12.949503 20898 raft_consensus.cc:482] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:12.949628 20898 raft_consensus.cc:3037] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.951886 20898 raft_consensus.cc:504] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.952510 20898 leader_election.cc:288] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099), 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:12.952910 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05e8456105e4459a4ea5695958fa95d" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
I20260810 19:20:12.953166 20257 raft_consensus.cc:3037] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.953137 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05e8456105e4459a4ea5695958fa95d" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40"
I20260810 19:20:12.953349 20741 raft_consensus.cc:3037] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:12.956100 20257 raft_consensus.cc:2454] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 1.
I20260810 19:20:12.956168 20741 raft_consensus.cc:2454] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 1.
I20260810 19:20:12.956549 20380 leader_election.cc:302] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee45652e5a84b43aa8f6c7fdc16a15c, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:12.956851 20898 raft_consensus.cc:2781] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:12.957013 20898 raft_consensus.cc:686] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Running, Role: LEADER
I20260810 19:20:12.957281 20898 consensus_queue.cc:237] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:12.959143 20874 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: TxnStatusTablet state changed. Reason: New leader 7da38ef950dc4a36a1b3bb81e9c08ed9. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:12.959450 20874 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:20:12.960206 20941 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 19:20:12.960441 20941 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 19:20:12.960870 19983 catalog_manager.cc:5495] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } health_report { overall_health: UNKNOWN } } }
I20260810 19:20:13.008611 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.003118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45027 (local address 127.0.0.1:47702)
0810 19:20:13.003321 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.003338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.003455 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.003833 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.003836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.003846 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:20:13.004016 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.004023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.005416 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.005419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.006608 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.006626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.007597 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.007600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.007738 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.007793 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.008195 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.008205 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.008259 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.008365 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:20:13.008703 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.003201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.7.43.2:45027)
0810 19:20:13.003412 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.003419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.003442 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:20:13.003614 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.003619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.003711 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.003814 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.004133 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.005298 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.006760 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.007454 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.007579 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.007976 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.008107 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.008200 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.008285 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.008366 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:20:13.012573 20257 raft_consensus.cc:1261] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Refusing update from remote peer 7da38ef950dc4a36a1b3bb81e9c08ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.012650 20741 raft_consensus.cc:1261] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Refusing update from remote peer 7da38ef950dc4a36a1b3bb81e9c08ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.013219 20874 consensus_queue.cc:1035] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.013787 20898 consensus_queue.cc:1035] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.017042 20876 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: TxnStatusTablet state changed. Reason: New leader 7da38ef950dc4a36a1b3bb81e9c08ed9. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:13.017314 20876 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:13.020697 20898 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:13.020993 20898 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:20:13.022522 20889 mvcc.cc:205] Tried to move back new op lower bound from 7317051854893158400 to 7317051854675255296. Current Snapshot: MvccSnapshot[applied={T|T < 7317051854893158400}]
I20260810 19:20:13.022794 20876 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:13.023056 20876 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 11574d84791a4ff8b2286314bad5dc40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:13.029361 20875 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: TxnStatusTablet state changed. Reason: New leader 7da38ef950dc4a36a1b3bb81e9c08ed9. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:13.030714 20875 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:13.031214 20874 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:13.031512 20874 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 7da38ef950dc4a36a1b3bb81e9c08ed9: This TxnStatusTablet replica's current role is: LEADER
I20260810 19:20:13.042934 20875 tablet_replica.cc:439] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } } }
I20260810 19:20:13.043198 20875 tablet_replica.cc:442] T f05e8456105e4459a4ea5695958fa95d P 3ee45652e5a84b43aa8f6c7fdc16a15c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 19:20:13.051302 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.030265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.7.43.3:38239)
0810 19:20:13.042915 (+ 12650us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.042922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.042951 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:20:13.043064 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.043070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.043195 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.043385 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.045029 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.046247 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.048236 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.048940 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.049069 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.049615 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.050377 (+ 762us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.050556 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.050653 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.050743 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12502,"thread_start_us":13269,"threads_started":1}
I20260810 19:20:13.068961 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.062936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:44853 (local address 127.7.43.2:45027)
0810 19:20:13.063235 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.063241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.063345 (+ 104us) server_negotiation.cc:407] Connection header received
0810 19:20:13.063538 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.063542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.063614 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.063696 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.064268 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.065059 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.067131 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.067678 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.067777 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.068287 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.068424 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.068501 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.068641 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.068708 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 19:20:13.069159 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.062602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45027 (local address 127.7.43.3:44853)
0810 19:20:13.063234 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.063254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.063412 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.063726 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.063731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.063744 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:20:13.064108 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.064120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.065185 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.065190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.066970 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.066986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.067855 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.067862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.068041 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.068115 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.068519 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.068538 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.068620 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.068783 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":537,"threads_started":1}
I20260810 19:20:13.079119 20504 raft_consensus.cc:1261] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Refusing update from remote peer 11574d84791a4ff8b2286314bad5dc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.079131 20257 raft_consensus.cc:1261] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Refusing update from remote peer 11574d84791a4ff8b2286314bad5dc40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.079772 20922 consensus_queue.cc:1035] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.080263 20922 consensus_queue.cc:1035] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.118678 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.112637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:36393 (local address 127.7.43.3:38239)
0810 19:20:13.112808 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.112814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.113004 (+ 190us) server_negotiation.cc:407] Connection header received
0810 19:20:13.113138 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.113143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.113228 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.113322 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.114025 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.115000 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.116924 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.117332 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.117439 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.117945 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.118047 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.118120 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.118276 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.118357 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:20:13.118811 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.112271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:38239 (local address 127.7.43.2:36393)
0810 19:20:13.112914 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.112930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.113042 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.113354 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.113357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.113366 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:20:13.113815 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.113835 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.115125 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.115131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.116782 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.116797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.117506 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.117534 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.117697 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.117761 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.118148 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.118173 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.118242 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.118388 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":293,"threads_started":1}
I20260810 19:20:13.173555 19980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:13.173831 19980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:13.177328 19980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Sending DeleteTablet for 3 replicas of tablet 9638021f693646bc87f9d3077dc875e3
I20260810 19:20:13.178229 20223 tablet_service.cc:1514] Processing DeleteTablet for tablet 9638021f693646bc87f9d3077dc875e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:57648
I20260810 19:20:13.178860 20455 tablet_service.cc:1514] Processing DeleteTablet for tablet 9638021f693646bc87f9d3077dc875e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260810 19:20:13.178887 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet 9638021f693646bc87f9d3077dc875e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:48068
I20260810 19:20:13.189879 20975 tablet_replica.cc:330] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: stopping tablet replica
I20260810 19:20:13.190155 20975 raft_consensus.cc:2229] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:13.190397 20975 raft_consensus.cc:2258] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.192836 20975 ts_tablet_manager.cc:1905] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.195024 20976 tablet_replica.cc:330] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: stopping tablet replica
I20260810 19:20:13.195823 20976 raft_consensus.cc:2229] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:13.196834 20976 raft_consensus.cc:2258] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.197463 20977 tablet_replica.cc:330] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: stopping tablet replica
I20260810 19:20:13.197701 20977 raft_consensus.cc:2229] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:13.198014 20977 raft_consensus.cc:2258] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.199287 20977 ts_tablet_manager.cc:1905] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.199882 20975 ts_tablet_manager.cc:1918] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:20:13.200017 20975 log.cc:1196] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/9638021f693646bc87f9d3077dc875e3
I20260810 19:20:13.200464 20975 ts_tablet_manager.cc:1939] T 9638021f693646bc87f9d3077dc875e3 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting consensus metadata
I20260810 19:20:13.201722 19959 catalog_manager.cc:4841] TS 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099): tablet 9638021f693646bc87f9d3077dc875e3 (table test-table [id=13e434bbbc3c409383639ed0c9e601d6]) successfully deleted
I20260810 19:20:13.202692 20976 ts_tablet_manager.cc:1905] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.210289 20976 ts_tablet_manager.cc:1918] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:20:13.210433 20976 log.cc:1196] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/9638021f693646bc87f9d3077dc875e3
I20260810 19:20:13.210867 20976 ts_tablet_manager.cc:1939] T 9638021f693646bc87f9d3077dc875e3 P 11574d84791a4ff8b2286314bad5dc40: Deleting consensus metadata
I20260810 19:20:13.211943 19958 catalog_manager.cc:4841] TS 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239): tablet 9638021f693646bc87f9d3077dc875e3 (table test-table [id=13e434bbbc3c409383639ed0c9e601d6]) successfully deleted
I20260810 19:20:13.215266 20977 ts_tablet_manager.cc:1918] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:20:13.215479 20977 log.cc:1196] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/9638021f693646bc87f9d3077dc875e3
I20260810 19:20:13.216051 20977 ts_tablet_manager.cc:1939] T 9638021f693646bc87f9d3077dc875e3 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting consensus metadata
I20260810 19:20:13.217289 19961 catalog_manager.cc:4841] TS 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027): tablet 9638021f693646bc87f9d3077dc875e3 (table test-table [id=13e434bbbc3c409383639ed0c9e601d6]) successfully deleted
Aug 10 19:20:13 dist-test-slave-7lbf krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 19:20:13 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:13 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:20:13.300925 20989 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:13.271400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.7.43.62:40505)
0810 19:20:13.287542 (+ 16142us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.287550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.287576 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:20:13.287648 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.287653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.287734 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.287865 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:13.289748 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.290866 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.292759 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.293411 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.293596 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.300612 (+ 7016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38818: BlockingRecv error: recv got EOF from 127.0.0.1:38818 (error 108)
Metrics: {"server-negotiator.queue_time_us":15986,"thread_start_us":16069,"threads_started":1}
Aug 10 19:20:13 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM
I20260810 19:20:13.330509 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.306769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.7.43.62:40505)
0810 19:20:13.306954 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.306961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.312495 (+ 5534us) server_negotiation.cc:407] Connection header received
0810 19:20:13.312676 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.312682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.312772 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.312890 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:13.314601 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.315762 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.317691 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.318334 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.318472 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.323550 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:13.323558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:13.323563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:13.323609 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:13.327448 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:13.328296 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:13.328302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:13.328306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:13.328476 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:13.328933 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:13.328938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:13.328942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:13.329298 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:13.329580 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.329925 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.330162 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:20:13.334726 19980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:13.335472 19980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:13.343554 20455 tablet_service.cc:1467] Processing CreateTablet for tablet 7933b22a8a934eeb9c5211d92c7f2b38 (DEFAULT_TABLE table=test-table [id=08d7c6e867f04516aa896459d0d5aaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:13.343703 20702 tablet_service.cc:1467] Processing CreateTablet for tablet 7933b22a8a934eeb9c5211d92c7f2b38 (DEFAULT_TABLE table=test-table [id=08d7c6e867f04516aa896459d0d5aaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:13.344055 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7933b22a8a934eeb9c5211d92c7f2b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:13.344174 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7933b22a8a934eeb9c5211d92c7f2b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:13.350338 20223 tablet_service.cc:1467] Processing CreateTablet for tablet 7933b22a8a934eeb9c5211d92c7f2b38 (DEFAULT_TABLE table=test-table [id=08d7c6e867f04516aa896459d0d5aaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:13.350814 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7933b22a8a934eeb9c5211d92c7f2b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:13.359637 21004 tablet_bootstrap.cc:492] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Bootstrap starting.
I20260810 19:20:13.362360 21004 tablet_bootstrap.cc:654] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:13.371290 21004 tablet_bootstrap.cc:492] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: No bootstrap required, opened a new log
I20260810 19:20:13.371567 21004 ts_tablet_manager.cc:1397] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260810 19:20:13.372561 21004 raft_consensus.cc:348] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.373001 21004 raft_consensus.cc:374] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:13.373164 21004 raft_consensus.cc:729] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11574d84791a4ff8b2286314bad5dc40, State: Initialized, Role: FOLLOWER
I20260810 19:20:13.373589 21004 consensus_queue.cc:260] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.374351 21004 ts_tablet_manager.cc:1428] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 19:20:13.377640 21009 tablet_bootstrap.cc:492] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Bootstrap starting.
I20260810 19:20:13.379683 21006 tablet_bootstrap.cc:492] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Bootstrap starting.
I20260810 19:20:13.382728 21006 tablet_bootstrap.cc:654] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:13.385064 21009 tablet_bootstrap.cc:654] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:13.387691 21006 tablet_bootstrap.cc:492] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: No bootstrap required, opened a new log
I20260810 19:20:13.389125 21006 ts_tablet_manager.cc:1397] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260810 19:20:13.390036 21006 raft_consensus.cc:348] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.390323 21006 raft_consensus.cc:374] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:13.390420 21006 raft_consensus.cc:729] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee45652e5a84b43aa8f6c7fdc16a15c, State: Initialized, Role: FOLLOWER
I20260810 19:20:13.390724 21006 consensus_queue.cc:260] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.391428 21006 ts_tablet_manager.cc:1428] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:13.394341 21009 tablet_bootstrap.cc:492] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: No bootstrap required, opened a new log
I20260810 19:20:13.394532 21009 ts_tablet_manager.cc:1397] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260810 19:20:13.395504 21009 raft_consensus.cc:348] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.395781 21009 raft_consensus.cc:374] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:13.395896 21009 raft_consensus.cc:729] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Initialized, Role: FOLLOWER
I20260810 19:20:13.396211 21009 consensus_queue.cc:260] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.400493 21009 ts_tablet_manager.cc:1428] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260810 19:20:13.510186 20881 raft_consensus.cc:482] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:13.510453 20881 raft_consensus.cc:504] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.511394 20881 leader_election.cc:288] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099), 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:13.512072 20256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7933b22a8a934eeb9c5211d92c7f2b38" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" is_pre_election: true
I20260810 19:20:13.512392 20256 raft_consensus.cc:2454] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 0.
I20260810 19:20:13.512948 20380 leader_election.cc:302] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee45652e5a84b43aa8f6c7fdc16a15c, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:13.512941 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7933b22a8a934eeb9c5211d92c7f2b38" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40" is_pre_election: true
I20260810 19:20:13.513254 20741 raft_consensus.cc:2454] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 0.
I20260810 19:20:13.514114 20881 raft_consensus.cc:2781] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:13.514235 20881 raft_consensus.cc:482] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:13.514353 20881 raft_consensus.cc:3037] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:13.517449 20881 raft_consensus.cc:504] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.518167 20881 leader_election.cc:288] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099), 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:13.518637 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7933b22a8a934eeb9c5211d92c7f2b38" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40"
I20260810 19:20:13.518632 20256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7933b22a8a934eeb9c5211d92c7f2b38" candidate_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c"
I20260810 19:20:13.518824 20741 raft_consensus.cc:3037] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:13.518848 20256 raft_consensus.cc:3037] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:13.520941 20741 raft_consensus.cc:2454] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 1.
I20260810 19:20:13.521332 20376 leader_election.cc:302] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:13.521941 20256 raft_consensus.cc:2454] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da38ef950dc4a36a1b3bb81e9c08ed9 in term 1.
I20260810 19:20:13.522279 20881 raft_consensus.cc:2781] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:13.522470 20881 raft_consensus.cc:686] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Running, Role: LEADER
I20260810 19:20:13.522758 20881 consensus_queue.cc:237] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.525799 19980 catalog_manager.cc:5495] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } health_report { overall_health: UNKNOWN } } }
I20260810 19:20:13.560336 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.555521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.7.43.2:45027)
0810 19:20:13.555689 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.555695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.555894 (+ 199us) server_negotiation.cc:407] Connection header received
0810 19:20:13.556081 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.556086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.556165 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.556240 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.556629 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.557322 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.558668 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.559095 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.559203 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.559640 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.559767 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.559840 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.559992 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.560070 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 19:20:13.562844 20256 raft_consensus.cc:1261] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Refusing update from remote peer 7da38ef950dc4a36a1b3bb81e9c08ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.562919 20741 raft_consensus.cc:1261] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Refusing update from remote peer 7da38ef950dc4a36a1b3bb81e9c08ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.563529 20898 consensus_queue.cc:1035] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.563985 20898 consensus_queue.cc:1035] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.578850 19983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:13.579078 19983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:13.581967 19983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Sending DeleteTablet for 3 replicas of tablet 7933b22a8a934eeb9c5211d92c7f2b38
I20260810 19:20:13.582639 20223 tablet_service.cc:1514] Processing DeleteTablet for tablet 7933b22a8a934eeb9c5211d92c7f2b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:57648
I20260810 19:20:13.582831 20455 tablet_service.cc:1514] Processing DeleteTablet for tablet 7933b22a8a934eeb9c5211d92c7f2b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260810 19:20:13.583105 20977 tablet_replica.cc:330] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: stopping tablet replica
I20260810 19:20:13.583114 20975 tablet_replica.cc:330] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: stopping tablet replica
I20260810 19:20:13.583312 20977 raft_consensus.cc:2229] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:13.583314 20975 raft_consensus.cc:2229] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:13.583539 20975 raft_consensus.cc:2258] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.583606 20977 raft_consensus.cc:2258] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.584654 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet 7933b22a8a934eeb9c5211d92c7f2b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:48068
I20260810 19:20:13.584733 20975 ts_tablet_manager.cc:1905] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.584928 20976 tablet_replica.cc:330] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: stopping tablet replica
I20260810 19:20:13.585116 20976 raft_consensus.cc:2229] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:13.585289 20976 raft_consensus.cc:2258] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.584753 19983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:13.586020 19983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:13.586385 20976 ts_tablet_manager.cc:1905] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.591748 20975 ts_tablet_manager.cc:1918] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:13.591923 20975 log.cc:1196] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/7933b22a8a934eeb9c5211d92c7f2b38
I20260810 19:20:13.593190 20223 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0e04e7eeff413c83f03e34b912689d (DEFAULT_TABLE table=test-table [id=bf71d209b3094e5497f6abf1247a113d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:13.593626 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0e04e7eeff413c83f03e34b912689d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:13.593621 20977 ts_tablet_manager.cc:1905] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.597787 20702 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0e04e7eeff413c83f03e34b912689d (DEFAULT_TABLE table=test-table [id=bf71d209b3094e5497f6abf1247a113d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:13.598131 20455 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0e04e7eeff413c83f03e34b912689d (DEFAULT_TABLE table=test-table [id=bf71d209b3094e5497f6abf1247a113d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:13.598289 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0e04e7eeff413c83f03e34b912689d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:13.598652 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0e04e7eeff413c83f03e34b912689d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:13.607816 20976 ts_tablet_manager.cc:1918] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:13.608026 20976 log.cc:1196] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/7933b22a8a934eeb9c5211d92c7f2b38
I20260810 19:20:13.608060 21006 tablet_bootstrap.cc:492] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Bootstrap starting.
I20260810 19:20:13.608518 20976 ts_tablet_manager.cc:1939] T 7933b22a8a934eeb9c5211d92c7f2b38 P 11574d84791a4ff8b2286314bad5dc40: Deleting consensus metadata
I20260810 19:20:13.609721 19958 catalog_manager.cc:4841] TS 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239): tablet 7933b22a8a934eeb9c5211d92c7f2b38 (table test-table [id=08d7c6e867f04516aa896459d0d5aaf3]) successfully deleted
I20260810 19:20:13.610608 21004 tablet_bootstrap.cc:492] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Bootstrap starting.
I20260810 19:20:13.612349 21004 tablet_bootstrap.cc:654] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:13.612433 20977 ts_tablet_manager.cc:1918] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:13.612603 20977 log.cc:1196] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/7933b22a8a934eeb9c5211d92c7f2b38
I20260810 19:20:13.613062 20977 ts_tablet_manager.cc:1939] T 7933b22a8a934eeb9c5211d92c7f2b38 P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting consensus metadata
I20260810 19:20:13.613744 21009 tablet_bootstrap.cc:492] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Bootstrap starting.
I20260810 19:20:13.615489 21009 tablet_bootstrap.cc:654] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:13.616408 20975 ts_tablet_manager.cc:1939] T 7933b22a8a934eeb9c5211d92c7f2b38 P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting consensus metadata
I20260810 19:20:13.617576 21006 tablet_bootstrap.cc:654] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:13.617599 19959 catalog_manager.cc:4841] TS 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099): tablet 7933b22a8a934eeb9c5211d92c7f2b38 (table test-table [id=08d7c6e867f04516aa896459d0d5aaf3]) successfully deleted
I20260810 19:20:13.619253 19961 catalog_manager.cc:4841] TS 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027): tablet 7933b22a8a934eeb9c5211d92c7f2b38 (table test-table [id=08d7c6e867f04516aa896459d0d5aaf3]) successfully deleted
I20260810 19:20:13.620893 21009 tablet_bootstrap.cc:492] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: No bootstrap required, opened a new log
I20260810 19:20:13.621047 21009 ts_tablet_manager.cc:1397] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260810 19:20:13.621865 21009 raft_consensus.cc:348] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.622195 21009 raft_consensus.cc:374] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:13.622308 21009 raft_consensus.cc:729] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da38ef950dc4a36a1b3bb81e9c08ed9, State: Initialized, Role: FOLLOWER
I20260810 19:20:13.622620 21009 consensus_queue.cc:260] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.622962 21006 tablet_bootstrap.cc:492] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: No bootstrap required, opened a new log
I20260810 19:20:13.623186 21006 ts_tablet_manager.cc:1397] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260810 19:20:13.623381 21009 ts_tablet_manager.cc:1428] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:13.624763 21004 tablet_bootstrap.cc:492] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: No bootstrap required, opened a new log
I20260810 19:20:13.624982 21004 ts_tablet_manager.cc:1397] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260810 19:20:13.625669 21006 raft_consensus.cc:348] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.626029 21006 raft_consensus.cc:374] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:13.625893 21004 raft_consensus.cc:348] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.626202 21004 raft_consensus.cc:374] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:13.626220 21006 raft_consensus.cc:729] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee45652e5a84b43aa8f6c7fdc16a15c, State: Initialized, Role: FOLLOWER
I20260810 19:20:13.626304 21004 raft_consensus.cc:729] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11574d84791a4ff8b2286314bad5dc40, State: Initialized, Role: FOLLOWER
I20260810 19:20:13.626624 21004 consensus_queue.cc:260] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.627220 21004 ts_tablet_manager.cc:1428] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 19:20:13.627074 21006 consensus_queue.cc:260] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.627794 21006 ts_tablet_manager.cc:1428] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 19:20:13.634467 20875 raft_consensus.cc:482] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:13.634651 20875 raft_consensus.cc:504] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.636314 20875 leader_election.cc:288] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027), 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:13.652593 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.637548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:52975 (local address 127.7.43.2:45027)
0810 19:20:13.637725 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.637732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.637751 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:13.637822 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.637827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.637901 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.637995 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:13.645219 (+ 7224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.646296 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.650101 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.651439 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.651538 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.651644 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.652159 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.652297 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:20:13.653452 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.636540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45027 (local address 127.7.43.1:52975)
0810 19:20:13.637202 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.637222 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.637348 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.643968 (+ 6620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.643974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.644012 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:13.644363 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.644380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.646521 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.646527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.648664 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.648682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.651789 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.651794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.651983 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.652015 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.652933 (+ 918us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.653095 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":1353,"threads_started":1}
I20260810 19:20:13.654276 20504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0e04e7eeff413c83f03e34b912689d" candidate_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" is_pre_election: true
I20260810 19:20:13.654543 20504 raft_consensus.cc:2454] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee45652e5a84b43aa8f6c7fdc16a15c in term 0.
I20260810 19:20:13.655184 20144 leader_election.cc:302] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee45652e5a84b43aa8f6c7fdc16a15c, 7da38ef950dc4a36a1b3bb81e9c08ed9; no voters:
I20260810 19:20:13.655684 20875 raft_consensus.cc:2781] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:13.655810 20875 raft_consensus.cc:482] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:13.655905 20875 raft_consensus.cc:3037] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:13.657238 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.637708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:54273 (local address 127.7.43.3:38239)
0810 19:20:13.649472 (+ 11764us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.649479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.649886 (+ 407us) server_negotiation.cc:407] Connection header received
0810 19:20:13.650069 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.650075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.650161 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.650269 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:13.651146 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.652881 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.655024 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.656119 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.656251 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.656379 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.656751 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.656910 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11616,"thread_start_us":11514,"threads_started":1}
I20260810 19:20:13.657265 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.636989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:38239 (local address 127.7.43.1:54273)
0810 19:20:13.649770 (+ 12781us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.649791 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.649919 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.650471 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.650477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.650507 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:13.650939 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.650951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.653064 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.653071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.654769 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.654800 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.656351 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.656359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.656560 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.656588 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.656695 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.656842 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12648,"thread_start_us":12520,"threads_started":1}
I20260810 19:20:13.658109 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0e04e7eeff413c83f03e34b912689d" candidate_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40" is_pre_election: true
I20260810 19:20:13.658375 20741 raft_consensus.cc:2454] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ee45652e5a84b43aa8f6c7fdc16a15c in term 0.
I20260810 19:20:13.659207 20875 raft_consensus.cc:504] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.659871 20875 leader_election.cc:288] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [CANDIDATE]: Term 1 election: Requested vote from peers 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027), 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239)
I20260810 19:20:13.660344 20741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0e04e7eeff413c83f03e34b912689d" candidate_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11574d84791a4ff8b2286314bad5dc40"
I20260810 19:20:13.660481 20504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0e04e7eeff413c83f03e34b912689d" candidate_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9"
I20260810 19:20:13.660542 20741 raft_consensus.cc:3037] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:13.660699 20504 raft_consensus.cc:3037] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:13.663272 20741 raft_consensus.cc:2454] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee45652e5a84b43aa8f6c7fdc16a15c in term 1.
I20260810 19:20:13.663515 20504 raft_consensus.cc:2454] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ee45652e5a84b43aa8f6c7fdc16a15c in term 1.
I20260810 19:20:13.663820 20140 leader_election.cc:302] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11574d84791a4ff8b2286314bad5dc40, 3ee45652e5a84b43aa8f6c7fdc16a15c; no voters:
I20260810 19:20:13.664227 20875 raft_consensus.cc:2781] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:13.664652 20875 raft_consensus.cc:686] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 LEADER]: Becoming Leader. State: Replica: 3ee45652e5a84b43aa8f6c7fdc16a15c, State: Running, Role: LEADER
I20260810 19:20:13.665305 20875 consensus_queue.cc:237] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } }
I20260810 19:20:13.668918 19983 catalog_manager.cc:5495] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1). New cstate: current_term: 1 leader_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee45652e5a84b43aa8f6c7fdc16a15c" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 } health_report { overall_health: UNKNOWN } } }
I20260810 19:20:13.694177 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.688947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.7.43.1:38099)
0810 19:20:13.689485 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.689490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.689508 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:20:13.689625 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.689628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.689698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.689835 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.690172 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.690973 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.692167 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.692568 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.692653 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.693047 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.693668 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.693783 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.693826 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.693897 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":291,"threads_started":1}
I20260810 19:20:13.708757 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.703836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:46237 (local address 127.7.43.2:45027)
0810 19:20:13.704074 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.704081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.704440 (+ 359us) server_negotiation.cc:407] Connection header received
0810 19:20:13.704607 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.704611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.704676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.704748 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.705228 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.705984 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.707307 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.707720 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.707840 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.708244 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.708343 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.708407 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.708492 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.708556 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:20:13.708853 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.703679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45027 (local address 127.7.43.1:46237)
0810 19:20:13.704338 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.704353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.704484 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.704775 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.704779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.704791 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:20:13.705095 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.705104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.706113 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.706117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.707173 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.707186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.707901 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.707906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.708036 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.708108 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.708411 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.708418 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.708472 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.708568 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":365,"threads_started":1}
I20260810 19:20:13.716869 20741 raft_consensus.cc:1261] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Refusing update from remote peer 3ee45652e5a84b43aa8f6c7fdc16a15c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.716869 20504 raft_consensus.cc:1261] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Refusing update from remote peer 3ee45652e5a84b43aa8f6c7fdc16a15c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:13.717468 21033 consensus_queue.cc:1035] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da38ef950dc4a36a1b3bb81e9c08ed9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.717841 20875 consensus_queue.cc:1035] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11574d84791a4ff8b2286314bad5dc40" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:13.722470 20888 mvcc.cc:205] Tried to move back new op lower bound from 7317051857778737152 to 7317051857577492480. Current Snapshot: MvccSnapshot[applied={T|T < 7317051857778737152}]
I20260810 19:20:13.741858 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.736618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:34099 (local address 127.7.43.1:38099)
0810 19:20:13.736794 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.736800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.737275 (+ 475us) server_negotiation.cc:407] Connection header received
0810 19:20:13.737390 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.737393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.737453 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.737546 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:13.738068 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.738762 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.740270 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.740716 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.740832 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.741166 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.741257 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.741328 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.741450 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.741557 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:20:13.741925 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.736604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38099 (local address 127.7.43.2:34099)
0810 19:20:13.737184 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:13.737200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:13.737301 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:13.737565 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:13.737570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:13.737583 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 19:20:13.737835 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.737847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.738897 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.738903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.740141 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:13.740154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.740853 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.740859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:13.740995 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.741032 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 19:20:13.741343 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 19:20:13.741356 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 19:20:13.741421 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 19:20:13.741585 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":355,"threads_started":1}
I20260810 19:20:13.774348 19983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:13.774610 19983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:13.777621 19983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6f3c6d8b3cc41c1b9634e531f97a6f1: Sending DeleteTablet for 3 replicas of tablet 7b0e04e7eeff413c83f03e34b912689d
I20260810 19:20:13.778439 20223 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0e04e7eeff413c83f03e34b912689d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:57648
I20260810 19:20:13.778533 20455 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0e04e7eeff413c83f03e34b912689d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260810 19:20:13.778646 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0e04e7eeff413c83f03e34b912689d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:13 UTC) from {username='oryx', principal='oryx/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:48068
I20260810 19:20:13.778760 20975 tablet_replica.cc:330] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: stopping tablet replica
I20260810 19:20:13.778848 20977 tablet_replica.cc:330] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: stopping tablet replica
I20260810 19:20:13.778926 20976 tablet_replica.cc:330] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: stopping tablet replica
I20260810 19:20:13.778960 20975 raft_consensus.cc:2229] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:13.779026 20977 raft_consensus.cc:2229] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:13.779081 20976 raft_consensus.cc:2229] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:13.779234 20977 raft_consensus.cc:2258] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.779286 20976 raft_consensus.cc:2258] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.779493 20975 raft_consensus.cc:2258] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:13.780437 20976 ts_tablet_manager.cc:1905] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.780449 20977 ts_tablet_manager.cc:1905] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.780823 20975 ts_tablet_manager.cc:1905] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:13.787015 20977 ts_tablet_manager.cc:1918] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:20:13.787159 20977 log.cc:1196] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/7b0e04e7eeff413c83f03e34b912689d
I20260810 19:20:13.787627 20977 ts_tablet_manager.cc:1939] T 7b0e04e7eeff413c83f03e34b912689d P 7da38ef950dc4a36a1b3bb81e9c08ed9: Deleting consensus metadata
I20260810 19:20:13.787777 20975 ts_tablet_manager.cc:1918] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:20:13.788026 20975 log.cc:1196] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/7b0e04e7eeff413c83f03e34b912689d
I20260810 19:20:13.788565 20975 ts_tablet_manager.cc:1939] T 7b0e04e7eeff413c83f03e34b912689d P 3ee45652e5a84b43aa8f6c7fdc16a15c: Deleting consensus metadata
I20260810 19:20:13.788730 19961 catalog_manager.cc:4841] TS 7da38ef950dc4a36a1b3bb81e9c08ed9 (127.7.43.2:45027): tablet 7b0e04e7eeff413c83f03e34b912689d (table test-table [id=bf71d209b3094e5497f6abf1247a113d]) successfully deleted
I20260810 19:20:13.789422 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 20117
I20260810 19:20:13.790184 19959 catalog_manager.cc:4841] TS 3ee45652e5a84b43aa8f6c7fdc16a15c (127.7.43.1:38099): tablet 7b0e04e7eeff413c83f03e34b912689d (table test-table [id=bf71d209b3094e5497f6abf1247a113d]) successfully deleted
I20260810 19:20:13.790757 20976 ts_tablet_manager.cc:1918] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 19:20:13.790899 20976 log.cc:1196] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/7b0e04e7eeff413c83f03e34b912689d
I20260810 19:20:13.791280 20976 ts_tablet_manager.cc:1939] T 7b0e04e7eeff413c83f03e34b912689d P 11574d84791a4ff8b2286314bad5dc40: Deleting consensus metadata
I20260810 19:20:13.792341 19958 catalog_manager.cc:4841] TS 11574d84791a4ff8b2286314bad5dc40 (127.7.43.3:38239): tablet 7b0e04e7eeff413c83f03e34b912689d (table test-table [id=bf71d209b3094e5497f6abf1247a113d]) successfully deleted
Aug 10 19:20:13 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.1@KRBTEST.COM
I20260810 19:20:13.804986 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.790732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.7.43.1:38099)
0810 19:20:13.790908 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.790917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.790936 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:13.791041 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.791047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.791157 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.791237 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:13.792604 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.793392 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.794633 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.795056 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.795166 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.799058 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:13.799070 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:13.799090 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:13.799122 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:13.802258 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:13.802885 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:13.802892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:13.802894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:13.803049 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:13.803408 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:13.803413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:13.803417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:13.803796 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:13.804113 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.804397 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.804616 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:20:13.805707 20328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:13.911305 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 20117
I20260810 19:20:13.939368 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 20342
Aug 10 19:20:13 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.2@KRBTEST.COM
I20260810 19:20:13.952952 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:13.940071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.7.43.2:45027)
0810 19:20:13.940380 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:13.940386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:13.940405 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:13.940488 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:13.940493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:13.940579 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:13.940687 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:13.942039 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.942761 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.943903 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:13.944262 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:13.944350 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:13.948151 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:13.948163 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:13.948179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:13.948224 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:13.950807 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:13.951370 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:13.951377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:13.951379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:13.951517 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:13.951782 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:13.951785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:13.951787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:13.952102 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:13.952334 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:13.952530 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 19:20:13.952687 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260810 19:20:13.953599 20570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:14.097555 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 20342
I20260810 19:20:14.136123 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 20585
Aug 10 19:20:14 dist-test-slave-7lbf krb5kdc[19910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.43.3@KRBTEST.COM
I20260810 19:20:14.154290 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:14.136974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.7.43.3:38239)
0810 19:20:14.137161 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:14.137165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:14.137187 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:20:14.137386 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:14.137392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:14.137484 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:14.137616 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:14.139416 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:14.140157 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:14.142102 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:14.142515 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:14.142610 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:14.148406 (+ 5796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:14.148423 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:14.148442 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:14.148490 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:14.151372 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:14.152267 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:14.152275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:14.152278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:14.152439 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:14.152839 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:14.152842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:14.152844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:14.153212 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:14.153464 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:14.153852 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 19:20:14.154041 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:20:14.154935 20806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:14.277841 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 20585
I20260810 19:20:14.307497 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 19932
I20260810 19:20:14.308719 20068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:14.423559 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 19932
2026-08-10T19:20:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5997 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:20:14 dist-test-slave-7lbf krb5kdc[21064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:20:14 dist-test-slave-7lbf krb5kdc[21064](info): set up 2 sockets
Aug 10 19:20:14 dist-test-slave-7lbf krb5kdc[21064](info): commencing operation
krb5kdc: starting...
W20260810 19:20:15.674239 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.006s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:20:15 dist-test-slave-7lbf krb5kdc[21064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:20:15.944520 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:34877
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:39745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:16.288707 21090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:16.289022 21090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:16.289170 21090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:16.296429 21090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:16.296546 21090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:16.296633 21090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:16.296716 21090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:16.296801 21090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:16.306734 21090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:39745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:16.308027 21090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:16.309417 21090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:16.321436 21097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:16.321482 21098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:16.321687 21109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:16.322345 21090 server_base.cc:1034] running on GCE node
I20260810 19:20:16.323087 21090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:16.324527 21090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:16.325729 21090 hybrid_clock.cc:648] HybridClock initialized: now 1786389616325682 us; error 29 us; skew 500 ppm
I20260810 19:20:16.326059 21090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T19:20:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1965 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:20:16 dist-test-slave-7lbf krb5kdc[21119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:20:16 dist-test-slave-7lbf krb5kdc[21119](info): set up 2 sockets
Aug 10 19:20:16 dist-test-slave-7lbf krb5kdc[21119](info): commencing operation
krb5kdc: starting...
W20260810 19:20:17.673946 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:20:17 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:20:17.938423 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43247
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:18.267670 21151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:18.267992 21151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:18.268136 21151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:18.275383 21151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:18.275537 21151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:18.275620 21151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:18.275709 21151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:18.275772 21151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:18.285688 21151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:18.286998 21151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:18.288298 21151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:18.299856 21158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:18.299902 21162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:18.299944 21159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:18.301074 21151 server_base.cc:1034] running on GCE node
I20260810 19:20:18.301859 21151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:18.303228 21151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:18.304431 21151 hybrid_clock.cc:648] HybridClock initialized: now 1786389618304368 us; error 49 us; skew 500 ppm
I20260810 19:20:18.304765 21151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:18 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:18.310626 21151 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:20:18.312106 21151 webserver.cc:460] Webserver started at http://127.7.43.62:41313/ using document root <none> and password file <none>
I20260810 19:20:18.312788 21151 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:18.312901 21151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:18.313216 21151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:18.316262 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "ea042caf470d45b3a873201baa26c217"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "0b498ebacf72c2fd53a60cf714f1d58e"
server_key_iv: "bdd22bc95b6217a3ba99c921954208a7"
server_key_version: "encryptionkey@0"
I20260810 19:20:18.317049 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "ea042caf470d45b3a873201baa26c217"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "0b498ebacf72c2fd53a60cf714f1d58e"
server_key_iv: "bdd22bc95b6217a3ba99c921954208a7"
server_key_version: "encryptionkey@0"
I20260810 19:20:18.322314 21151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:20:18.325924 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:18.327339 21151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:20:18.327522 21151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "ea042caf470d45b3a873201baa26c217"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "0b498ebacf72c2fd53a60cf714f1d58e"
server_key_iv: "bdd22bc95b6217a3ba99c921954208a7"
server_key_version: "encryptionkey@0"
I20260810 19:20:18.327718 21151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:18.359975 21151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:18.364229 21151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:18.364502 21151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:18.382310 21151 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:43247
I20260810 19:20:18.382387 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:43247 every 8 connection(s)
I20260810 19:20:18.383723 21151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:20:18.387485 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:18.393112 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 21151
I20260810 19:20:18.393369 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:20:18.394208 7340 external_mini_cluster.cc:1428] Setting key 2163a490e558e8d7798c26dd3edbffa4
I20260810 19:20:18.399012 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Bootstrap starting.
I20260810 19:20:18.402869 21259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:18.404235 21259 log.cc:824] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:18.407482 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: No bootstrap required, opened a new log
Aug 10 19:20:18 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:18.416607 21259 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:18.417078 21259 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:18.417187 21259 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea042caf470d45b3a873201baa26c217, State: Initialized, Role: FOLLOWER
I20260810 19:20:18.417910 21259 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:18.418220 21259 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:20:18.418409 21259 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:20:18.418617 21259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:18.419960 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:18.396529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.7.43.62:43247)
0810 19:20:18.397442 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:18.397459 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:18.397891 (+ 432us) server_negotiation.cc:407] Connection header received
0810 19:20:18.400279 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:18.400310 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:18.400849 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:18.401347 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:18.402697 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:18.404672 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:18.406249 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:18.407260 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:18.407435 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:18.411203 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:18.411218 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:18.411236 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:18.411291 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:18.415282 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:18.415941 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:18.415950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:18.415958 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:18.416165 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:18.416646 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:18.416650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:18.416653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:18.417378 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:18.417714 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:18.418125 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 19:20:18.418576 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":405,"threads_started":1}
I20260810 19:20:18.421375 21259 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:18.421885 21259 leader_election.cc:302] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea042caf470d45b3a873201baa26c217; no voters:
I20260810 19:20:18.422443 21259 leader_election.cc:288] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:20:18.422595 21267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:18.423388 21267 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 LEADER]: Becoming Leader. State: Replica: ea042caf470d45b3a873201baa26c217, State: Running, Role: LEADER
I20260810 19:20:18.423862 21267 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:18.424582 21259 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:18.426581 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea042caf470d45b3a873201baa26c217. Latest consensus state: current_term: 1 leader_uuid: "ea042caf470d45b3a873201baa26c217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } } }
I20260810 19:20:18.426645 21268 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea042caf470d45b3a873201baa26c217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } } }
I20260810 19:20:18.426972 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:18.426976 21268 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:18.435082 21276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:18.439322 21276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:18.446769 21276 catalog_manager.cc:1347] Generated new cluster ID: 2dcb880edd7b4bf8a1db475fa9705c31
I20260810 19:20:18.446875 21276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:18.455343 21276 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:20:18.457134 21276 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:18.471263 21276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Generated new TSK 0
I20260810 19:20:18.472252 21276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:20:18.547096 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43247
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:18.883019 21301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:18.883309 21301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:18.883441 21301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:18.890877 21301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:18.890998 21301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:18.891147 21301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:18.901091 21301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:18.902447 21301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:18.903796 21301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:18.916527 21307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:18.916481 21308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:18.916553 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:18.917341 21301 server_base.cc:1034] running on GCE node
I20260810 19:20:18.917944 21301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:18.918964 21301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:18.920189 21301 hybrid_clock.cc:648] HybridClock initialized: now 1786389618920135 us; error 49 us; skew 500 ppm
I20260810 19:20:18.920466 21301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:18 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:18.923417 21301 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:20:18.924836 21301 webserver.cc:460] Webserver started at http://127.7.43.1:35765/ using document root <none> and password file <none>
I20260810 19:20:18.925490 21301 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:18.925645 21301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:18.925940 21301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:18.929076 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "cbab3499b7534ac494488cf025b80c62"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "39f779389258082ecb59cd3e32b0589d"
server_key_iv: "0aed90564b29e53ca9abe3f19e05733e"
server_key_version: "encryptionkey@0"
I20260810 19:20:18.929845 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "cbab3499b7534ac494488cf025b80c62"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "39f779389258082ecb59cd3e32b0589d"
server_key_iv: "0aed90564b29e53ca9abe3f19e05733e"
server_key_version: "encryptionkey@0"
I20260810 19:20:18.934873 21301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:20:18.938097 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:18.939306 21301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:18.939460 21301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "cbab3499b7534ac494488cf025b80c62"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "39f779389258082ecb59cd3e32b0589d"
server_key_iv: "0aed90564b29e53ca9abe3f19e05733e"
server_key_version: "encryptionkey@0"
I20260810 19:20:18.939630 21301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:18.963330 21301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:18.966997 21301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:18.967252 21301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:18.968175 21301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:18.969506 21301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:18.969612 21301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:18.969728 21301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:18.969780 21301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:19.000562 21301 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:35235
I20260810 19:20:19.000612 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:35235 every 8 connection(s)
I20260810 19:20:19.001909 21301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:19.005129 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 21301
I20260810 19:20:19.005326 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:20:19.006011 7340 external_mini_cluster.cc:1428] Setting key 13dd5312b8722204e173e714189a72b7
Aug 10 19:20:19 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:19.022925 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:19.004476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:47051 (local address 127.7.43.62:43247)
0810 19:20:19.004960 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:19.004966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:19.006101 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 19:20:19.008143 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:19.008148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:19.008226 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:19.008342 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:19.009837 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.010865 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.012369 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.012897 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.013032 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:19.016676 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:19.016682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:19.016687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:19.016727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:19.019751 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:19.020661 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:19.020665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:19.020667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:19.020801 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:19.021267 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:19.021270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:19.021272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:19.021564 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:19.021682 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:19.022492 (+ 810us) server_negotiation.cc:299] Negotiation successful
0810 19:20:19.022660 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260810 19:20:19.024394 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:19.005040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:47051)
0810 19:20:19.005896 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:19.005964 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:19.007866 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:19.008531 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:19.008538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:19.009048 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:19.009668 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:19.009689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.011011 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.011015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:19.012229 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:19.012241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.013118 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.013122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:19.013342 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:19.014016 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:19.014039 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:19.016441 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:19.019896 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:19.019903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:19.019918 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:19.020524 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:19.020908 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:19.020912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:19.020916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:19.021162 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:19.021699 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:19.021707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:19.022112 (+ 405us) client_negotiation.cc:769] Sending connection context
0810 19:20:19.022409 (+ 297us) client_negotiation.cc:240] Negotiation successful
0810 19:20:19.022930 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":334,"threads_started":1}
I20260810 19:20:19.026023 21503 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43247
I20260810 19:20:19.026360 21503 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:19.027242 21503 heartbeater.cc:511] Master 127.7.43.62:43247 requested a full tablet report, sending...
I20260810 19:20:19.029631 21200 ts_manager.cc:194] Registered new tserver with Master: cbab3499b7534ac494488cf025b80c62 (127.7.43.1:35235)
I20260810 19:20:19.031644 21200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:47051
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:20:19.068526 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43247
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:19.402441 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:19.402796 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:19.402952 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:19.411001 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:19.411113 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:19.411231 21517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:19.421226 21517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:19.422525 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:19.423801 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:19.436578 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:19.436627 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:19.436620 21528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:19.437649 21517 server_base.cc:1034] running on GCE node
I20260810 19:20:19.438251 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:19.439121 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:19.440306 21517 hybrid_clock.cc:648] HybridClock initialized: now 1786389619440241 us; error 58 us; skew 500 ppm
I20260810 19:20:19.440608 21517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:19 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:19.443718 21517 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:20:19.445052 21517 webserver.cc:460] Webserver started at http://127.7.43.2:39411/ using document root <none> and password file <none>
I20260810 19:20:19.445742 21517 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:19.445845 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:19.446107 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:19.448885 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "cd9f26b2e68041e585c1a6bbe81f1bc3"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "3f37a0b3984071ff6dd89d782a28a9ee"
server_key_iv: "8224e234cd114f2b804c2e03d1df54f0"
server_key_version: "encryptionkey@0"
I20260810 19:20:19.449658 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "cd9f26b2e68041e585c1a6bbe81f1bc3"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "3f37a0b3984071ff6dd89d782a28a9ee"
server_key_iv: "8224e234cd114f2b804c2e03d1df54f0"
server_key_version: "encryptionkey@0"
I20260810 19:20:19.454547 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:20:19.457785 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:19.459028 21517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:20:19.459177 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "cd9f26b2e68041e585c1a6bbe81f1bc3"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "3f37a0b3984071ff6dd89d782a28a9ee"
server_key_iv: "8224e234cd114f2b804c2e03d1df54f0"
server_key_version: "encryptionkey@0"
I20260810 19:20:19.459350 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:19.490435 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:19.494302 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:19.494585 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:19.495533 21517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:19.496865 21517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:19.496945 21517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:19.497071 21517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:19.497124 21517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:19.527029 21517 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:46107
I20260810 19:20:19.527094 21732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:46107 every 8 connection(s)
I20260810 19:20:19.528311 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:19.532904 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 21517
I20260810 19:20:19.533113 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:20:19.533754 7340 external_mini_cluster.cc:1428] Setting key 151d8a99b26a5bd547f2b752000283c4
Aug 10 19:20:19 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:19.549319 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:19.530813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:37455 (local address 127.7.43.62:43247)
0810 19:20:19.531624 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:19.531631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:19.532542 (+ 911us) server_negotiation.cc:407] Connection header received
0810 19:20:19.534972 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:19.534977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:19.535040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:19.535125 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:19.536994 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.537736 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.539784 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.540199 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.540287 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:19.544051 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:19.544057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:19.544060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:19.544091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:19.546450 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:19.547115 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:19.547121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:19.547124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:19.547247 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:19.547651 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:19.547656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:19.547658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:19.547921 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:19.548061 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:19.548718 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 19:20:19.548961 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":565,"threads_started":1}
I20260810 19:20:19.550273 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:19.531473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:37455)
0810 19:20:19.532351 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:19.532406 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:19.534691 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:19.535388 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:19.535399 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:19.536044 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:19.536820 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:19.536846 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.537889 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.537894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:19.539661 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:19.539675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:19.540402 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:19.540408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:19.540680 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:19.541316 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:19.541344 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:19.543817 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:19.546577 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:19.546586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:19.546602 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:19.547003 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:19.547349 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:19.547354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:19.547358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:19.547547 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:19.548050 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:19.548057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:19.548368 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 19:20:19.548616 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 19:20:19.549110 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":378,"threads_started":1}
I20260810 19:20:19.551667 21733 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43247
I20260810 19:20:19.552011 21733 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:19.552870 21733 heartbeater.cc:511] Master 127.7.43.62:43247 requested a full tablet report, sending...
I20260810 19:20:19.554454 21200 ts_manager.cc:194] Registered new tserver with Master: cd9f26b2e68041e585c1a6bbe81f1bc3 (127.7.43.2:46107)
I20260810 19:20:19.555543 21200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:37455
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:20:19.596459 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43247
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:19.932271 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:19.932540 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:19.932638 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:19.939679 21740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:19.939785 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:19.939909 21740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:19.949316 21740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:19.950702 21740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:19.951951 21740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:19.964398 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:19.964481 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:19.964433 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:19.965711 21740 server_base.cc:1034] running on GCE node
I20260810 19:20:19.966271 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:19.967164 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:19.968377 21740 hybrid_clock.cc:648] HybridClock initialized: now 1786389619968327 us; error 33 us; skew 500 ppm
I20260810 19:20:19.968689 21740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:19 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:19.971936 21740 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:20:19.973376 21740 webserver.cc:460] Webserver started at http://127.7.43.3:41645/ using document root <none> and password file <none>
I20260810 19:20:19.974116 21740 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:19.974231 21740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:19.974532 21740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:19.977432 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "b281af738432475bbde220721cb47252"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "dd090d973bd0dd1fcc43da33e6d784ba"
server_key_iv: "dfe689bc9db253e30964d6eb7ea44b7b"
server_key_version: "encryptionkey@0"
I20260810 19:20:19.978255 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "b281af738432475bbde220721cb47252"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "dd090d973bd0dd1fcc43da33e6d784ba"
server_key_iv: "dfe689bc9db253e30964d6eb7ea44b7b"
server_key_version: "encryptionkey@0"
I20260810 19:20:19.983304 21740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:20:19.986625 21760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:19.987900 21740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:20:19.988044 21740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "b281af738432475bbde220721cb47252"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "dd090d973bd0dd1fcc43da33e6d784ba"
server_key_iv: "dfe689bc9db253e30964d6eb7ea44b7b"
server_key_version: "encryptionkey@0"
I20260810 19:20:19.988215 21740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:20.000631 21740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:20.004925 21740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:20.005203 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:20.006174 21740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:20.007481 21740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:20.007561 21740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:20.007682 21740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:20.007738 21740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:20.035382 21503 heartbeater.cc:503] Master 127.7.43.62:43247 was elected leader, sending a full tablet report...
I20260810 19:20:20.039289 21740 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:37555
I20260810 19:20:20.039309 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:37555 every 8 connection(s)
I20260810 19:20:20.040582 21740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:20.044396 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 21740
I20260810 19:20:20.044601 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:20:20.045125 7340 external_mini_cluster.cc:1428] Setting key f72327bd11faf735e669f019ccfdae90
Aug 10 19:20:20 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:20.060225 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:20.043130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:52817 (local address 127.7.43.62:43247)
0810 19:20:20.043334 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:20.043339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:20.044679 (+ 1340us) server_negotiation.cc:407] Connection header received
0810 19:20:20.046723 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:20.046728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:20.046811 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:20.046896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:20.048713 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.049439 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.051044 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.051459 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.051568 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:20.054784 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:20.054789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:20.054792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:20.054824 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:20.057294 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.058009 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.058012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.058014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.058131 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.058543 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.058546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.058548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.058828 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:20.058976 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:20.059678 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 19:20:20.059906 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:20:20.061122 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:20.043724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:52817)
0810 19:20:20.044515 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:20.044566 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:20.046464 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:20.047188 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:20.047201 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:20.047958 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:20.048550 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:20.048569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.049618 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.049622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:20.050901 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:20.050919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.051673 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.051681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:20.051897 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:20.052513 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:20.052536 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:20.054595 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:20.057421 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:20.057438 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:20.057454 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:20.057891 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:20.058230 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:20.058234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:20.058238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:20.058421 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:20.058957 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:20.058965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:20.059289 (+ 324us) client_negotiation.cc:769] Sending connection context
0810 19:20:20.059528 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 19:20:20.060032 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":325,"threads_started":1}
I20260810 19:20:20.062337 21957 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43247
I20260810 19:20:20.062669 21957 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:20.063349 21957 heartbeater.cc:511] Master 127.7.43.62:43247 requested a full tablet report, sending...
I20260810 19:20:20.064711 21200 ts_manager.cc:194] Registered new tserver with Master: b281af738432475bbde220721cb47252 (127.7.43.3:37555)
I20260810 19:20:20.065491 21200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:52817
I20260810 19:20:20.072299 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab.
Entry for principal oryx/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab.
WARNING: no policy specified for oryx/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab.
Entry for principal oryx/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab.
WARNING: no policy specified for oryx/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab.
Entry for principal oryx/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab.
I20260810 19:20:20.242560 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 21301
Aug 10 19:20:20 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:20:20.257956 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:20.243751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.7.43.1:35235)
0810 19:20:20.244419 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:20.244433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:20.244460 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:20:20.244530 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:20.244536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:20.244743 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:20.244897 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:20.246268 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.247299 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.248475 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.249027 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.249123 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:20.252866 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:20.252878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:20.252893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:20.252926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:20.255689 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.256262 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.256268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.256271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.256388 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.256661 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.256664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.256666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.256994 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:20.257210 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:20.257423 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 19:20:20.257673 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":338,"threads_started":1}
I20260810 19:20:20.259243 21497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:20.340339 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 21301
I20260810 19:20:20.361207 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 21517
Aug 10 19:20:20 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:20:20.377089 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:20.362085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.7.43.2:46107)
0810 19:20:20.362602 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:20.362608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:20.362635 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:20:20.362703 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:20.362709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:20.362916 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:20.363077 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:20.364338 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.365351 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.366678 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.367234 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.367347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:20.371552 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:20.371564 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:20.371579 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:20.371615 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:20.374339 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.375001 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.375008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.375011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.375135 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.375501 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.375505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.375507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.375855 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:20.376145 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:20.376459 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 19:20:20.376718 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":250,"threads_started":1}
I20260810 19:20:20.378374 21725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:20.462807 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 21517
I20260810 19:20:20.478942 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 21740
Aug 10 19:20:20 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:20:20.494341 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:20.479921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.7.43.3:37555)
0810 19:20:20.480559 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:20.480566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:20.480595 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:20:20.480653 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:20.480659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:20.480858 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:20.480985 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:20.482287 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.483311 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.484476 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:20.485020 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:20.485117 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:20.489267 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:20.489278 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:20.489293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:20.489325 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:20.491938 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.492657 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.492664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.492666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.492797 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:20.493058 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:20.493061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:20.493062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:20.493385 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:20.493675 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:20.493856 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 19:20:20.494050 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":326,"threads_started":1}
I20260810 19:20:20.495607 21952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:20.577905 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 21740
I20260810 19:20:20.599740 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 21151
I20260810 19:20:20.600965 21250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:20.702867 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 21151
I20260810 19:20:20.730351 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43247
--webserver_interface=127.7.43.62
--webserver_port=41313
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:21.063944 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:21.064275 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:21.064416 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:21.071739 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:21.071870 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:21.071949 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:21.072033 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:21.072095 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:21.082041 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=41313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:21.083341 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:21.084648 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:21.095798 21994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:21.095774 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:21.095923 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:21.097167 21985 server_base.cc:1034] running on GCE node
I20260810 19:20:21.097965 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:21.099313 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:21.100514 21985 hybrid_clock.cc:648] HybridClock initialized: now 1786389621100462 us; error 45 us; skew 500 ppm
I20260810 19:20:21.100852 21985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389621, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:21.103893 21985 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:20:21.105321 21985 webserver.cc:460] Webserver started at http://127.7.43.62:41313/ using document root <none> and password file <none>
I20260810 19:20:21.106040 21985 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:21.106158 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:21.111503 21985 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:20:21.114303 22011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:21.115631 21985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:20:21.115777 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "ea042caf470d45b3a873201baa26c217"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "0b498ebacf72c2fd53a60cf714f1d58e"
server_key_iv: "bdd22bc95b6217a3ba99c921954208a7"
server_key_version: "encryptionkey@0"
I20260810 19:20:21.116724 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:21.141419 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:21.145533 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:21.145762 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:21.163344 21985 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:43247
I20260810 19:20:21.163383 22100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:43247 every 8 connection(s)
I20260810 19:20:21.164816 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:20:21.167158 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 21985
I20260810 19:20:21.168105 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:35235
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:43247
--webserver_port=35765
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:20:21.173488 22103 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:20:21.177795 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Bootstrap starting.
I20260810 19:20:21.197007 22103 log.cc:824] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:21.206039 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:20:21.206606 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Bootstrap complete.
I20260810 19:20:21.215728 22103 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:21.216586 22103 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea042caf470d45b3a873201baa26c217, State: Initialized, Role: FOLLOWER
I20260810 19:20:21.217206 22103 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:21.217437 22103 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:20:21.217579 22103 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:20:21.217746 22103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:20:21.222327 22103 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:21.222916 22103 leader_election.cc:302] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea042caf470d45b3a873201baa26c217; no voters:
I20260810 19:20:21.223521 22103 leader_election.cc:288] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:20:21.223745 22108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:20:21.224677 22108 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [term 2 LEADER]: Becoming Leader. State: Replica: ea042caf470d45b3a873201baa26c217, State: Running, Role: LEADER
I20260810 19:20:21.225567 22103 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:21.225402 22108 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } }
I20260810 19:20:21.229408 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea042caf470d45b3a873201baa26c217. Latest consensus state: current_term: 2 leader_uuid: "ea042caf470d45b3a873201baa26c217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } } }
I20260810 19:20:21.229835 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:21.229669 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea042caf470d45b3a873201baa26c217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea042caf470d45b3a873201baa26c217" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43247 } } }
I20260810 19:20:21.229975 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:21.240271 22120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:21.246392 22120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:21.247834 22120 catalog_manager.cc:1259] Loaded cluster ID: 2dcb880edd7b4bf8a1db475fa9705c31
I20260810 19:20:21.247987 22120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:21.252080 22120 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:21.253582 22120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea042caf470d45b3a873201baa26c217: Loaded TSK: 0
I20260810 19:20:21.255142 22120 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:20:21.515061 22105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:21.515352 22105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:21.515467 22105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:21.522992 22105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:21.523121 22105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:21.523267 22105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:21.533248 22105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.1.keytab
--principal=oryx/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:35235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=35765
--tserver_master_addrs=127.7.43.62:43247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:21.534615 22105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:21.535910 22105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:21.551679 22151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:21.551759 22147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:21.552069 22146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:21.552809 22105 server_base.cc:1034] running on GCE node
I20260810 19:20:21.553608 22105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:21.554869 22105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:21.556113 22105 hybrid_clock.cc:648] HybridClock initialized: now 1786389621556057 us; error 40 us; skew 500 ppm
I20260810 19:20:21.556491 22105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389621, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:21.560173 22105 init.cc:377] Logged in from keytab as oryx/127.7.43.1@KRBTEST.COM (short username oryx)
I20260810 19:20:21.561707 22105 webserver.cc:460] Webserver started at http://127.7.43.1:35765/ using document root <none> and password file <none>
I20260810 19:20:21.562409 22105 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:21.562541 22105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:21.568260 22105 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 19:20:21.571213 22161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:21.572660 22105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:20:21.572831 22105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "cbab3499b7534ac494488cf025b80c62"
format_stamp: "Formatted at 2026-08-10 19:20:18 on dist-test-slave-7lbf"
server_key: "39f779389258082ecb59cd3e32b0589d"
server_key_iv: "0aed90564b29e53ca9abe3f19e05733e"
server_key_version: "encryptionkey@0"
I20260810 19:20:21.573848 22105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:21.587267 22105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:21.591476 22105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:21.591732 22105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:21.592702 22105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:21.594099 22105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:21.594197 22105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:21.594303 22105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:21.594373 22105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:21.624289 22105 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:35235
I20260810 19:20:21.624346 22359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:35235 every 8 connection(s)
I20260810 19:20:21.625771 22105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:21.627835 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 22105
I20260810 19:20:21.628618 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:46107
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:43247
--webserver_port=39411
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:21.644102 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:21.629110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:57273)
0810 19:20:21.630141 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:21.630178 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:21.631902 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:21.633291 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:21.633302 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:21.633945 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:21.634494 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:21.634513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.636080 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.636084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:21.637286 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:21.637299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.638548 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.638555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:21.638815 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:21.639521 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:21.639552 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:21.643176 (+ 3624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":606,"threads_started":1}
W20260810 19:20:21.644775 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:21.628959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:57273 (local address 127.7.43.62:43247)
0810 19:20:21.629654 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:21.629666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:21.630317 (+ 651us) server_negotiation.cc:407] Connection header received
0810 19:20:21.632435 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:21.632449 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:21.632771 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:21.633107 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:21.634683 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.635922 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.637426 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.638309 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.638469 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:21.643605 (+ 5136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:57273: BlockingRecv error: recv got EOF from 127.7.43.1:57273 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":232,"threads_started":1}
W20260810 19:20:21.644855 22361 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:43247 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.43.62:43247: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:21.653601 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:21.645409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:60609)
0810 19:20:21.645594 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:21.645614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:21.645712 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:21.646054 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:21.646059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:21.646477 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:21.646649 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:21.646660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.647651 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.647656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:21.648881 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:21.648896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.649589 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.649595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:21.649748 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:21.650292 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:21.650312 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:21.653302 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:20:21.653904 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:21.645660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:60609 (local address 127.7.43.62:43247)
0810 19:20:21.645799 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:21.645804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:21.645818 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:20:21.645863 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:21.645866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:21.645922 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:21.646002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:21.646770 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.647512 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.649022 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.649417 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.649541 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:21.653554 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:60609: BlockingRecv error: recv got EOF from 127.7.43.1:60609 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:21 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:21.662147 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:21.654231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:57753)
0810 19:20:21.654669 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:21.654690 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:21.654840 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:21.655161 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:21.655165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:21.655464 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:21.655603 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:21.655610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.656570 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.656576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:21.657470 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:21.657481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.658463 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.658467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:21.658588 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:21.659089 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:21.659106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:21.661845 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320}
W20260810 19:20:21.662384 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:21.654345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:57753 (local address 127.7.43.62:43247)
0810 19:20:21.654523 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:21.654528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:21.654755 (+ 227us) server_negotiation.cc:407] Connection header received
0810 19:20:21.654966 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:21.654970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:21.655039 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:21.655120 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:21.655730 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.656426 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.657625 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:21.658321 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:21.658462 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:21.662018 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:57753: BlockingRecv error: recv got EOF from 127.7.43.1:57753 (error 108)
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":385,"threads_started":1}
W20260810 19:20:21.662514 22361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:20:21.973711 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:21.973989 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:21.974092 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:21.981628 22370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:21.981734 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:21.981863 22370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:21.991724 22370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.2.keytab
--principal=oryx/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:46107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=39411
--tserver_master_addrs=127.7.43.62:43247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:21.993038 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:21.994392 22370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:22.007310 22379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:22.007275 22377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:22.007339 22381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:22.008075 22370 server_base.cc:1034] running on GCE node
I20260810 19:20:22.008688 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:22.009663 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:22.010912 22370 hybrid_clock.cc:648] HybridClock initialized: now 1786389622010849 us; error 55 us; skew 500 ppm
I20260810 19:20:22.011222 22370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389622, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:22.014312 22370 init.cc:377] Logged in from keytab as oryx/127.7.43.2@KRBTEST.COM (short username oryx)
I20260810 19:20:22.015784 22370 webserver.cc:460] Webserver started at http://127.7.43.2:39411/ using document root <none> and password file <none>
I20260810 19:20:22.016525 22370 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:22.016662 22370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:22.022225 22370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:20:22.024961 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:22.026432 22370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:20:22.026594 22370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "cd9f26b2e68041e585c1a6bbe81f1bc3"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "3f37a0b3984071ff6dd89d782a28a9ee"
server_key_iv: "8224e234cd114f2b804c2e03d1df54f0"
server_key_version: "encryptionkey@0"
I20260810 19:20:22.027542 22370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:22.056330 22370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:22.060704 22370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:22.060973 22370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:22.062031 22370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:22.063366 22370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:22.063462 22370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:22.063571 22370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:22.063642 22370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:22.097376 22370 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:46107
I20260810 19:20:22.097430 22589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:46107 every 8 connection(s)
I20260810 19:20:22.098938 22370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:22.102237 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 22370
I20260810 19:20:22.102898 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:37555
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:43247
--webserver_port=41645
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:20:22.115453 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.101497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:51395 (local address 127.7.43.62:43247)
0810 19:20:22.101688 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.101693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.102981 (+ 1288us) server_negotiation.cc:407] Connection header received
0810 19:20:22.104892 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.104898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.104986 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.105096 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.106743 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.107662 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.109332 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.109949 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.110084 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.115191 (+ 5107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:51395: BlockingRecv error: recv got EOF from 127.7.43.2:51395 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:20:22.116205 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.101952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:51395)
0810 19:20:22.102783 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.102840 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.104582 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.105343 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.105352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.105922 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.106547 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.106569 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.107813 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.107817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.109154 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.109167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.110157 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.110162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.110372 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.111026 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.111050 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.114979 (+ 3929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":336,"threads_started":1}
W20260810 19:20:22.116983 22591 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:43247 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.43.62:43247: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:22.126543 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.117587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:47219)
0810 19:20:22.117778 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.117798 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.117921 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.118255 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.118260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.118721 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.118902 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.118913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.119942 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.119947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.121275 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.121290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.122061 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.122066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.122238 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.122913 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.122936 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.126251 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 19:20:22.126672 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.117738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:47219 (local address 127.7.43.62:43247)
0810 19:20:22.117906 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.117911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.117929 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:20:22.118041 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.118045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.118106 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.118197 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.119028 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.119784 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.121421 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.121902 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.122011 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.126477 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:47219: BlockingRecv error: recv got EOF from 127.7.43.2:47219 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:22.134878 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.127456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:38015)
0810 19:20:22.127713 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.127728 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.127811 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.128091 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.128094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.128363 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.128515 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.128523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.129485 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.129489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.130452 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.130464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.131226 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.131231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.131396 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.131930 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.131947 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.134586 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260810 19:20:22.135035 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.127603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:38015 (local address 127.7.43.62:43247)
0810 19:20:22.127778 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.127783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.127802 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:20:22.127898 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.127902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.127973 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.128062 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.128658 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.129350 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.130617 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.131082 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.131189 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.134833 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:38015: BlockingRecv error: recv got EOF from 127.7.43.2:38015 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 19:20:22.135214 22591 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:20:22.441751 22595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:22.442091 22595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:22.442238 22595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:22.449390 22595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:22.449541 22595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:22.449704 22595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:22.459359 22595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/krb5kdc/oryx_127.7.43.3.keytab
--principal=oryx/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:37555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=41645
--tserver_master_addrs=127.7.43.62:43247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:22.460673 22595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:22.461990 22595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:22.474398 22604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:22.474363 22609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:22.474314 22607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:22.475175 22595 server_base.cc:1034] running on GCE node
I20260810 19:20:22.475747 22595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:22.476626 22595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:22.477816 22595 hybrid_clock.cc:648] HybridClock initialized: now 1786389622477771 us; error 34 us; skew 500 ppm
I20260810 19:20:22.478159 22595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389622, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:22.481194 22595 init.cc:377] Logged in from keytab as oryx/127.7.43.3@KRBTEST.COM (short username oryx)
I20260810 19:20:22.482656 22595 webserver.cc:460] Webserver started at http://127.7.43.3:41645/ using document root <none> and password file <none>
I20260810 19:20:22.483330 22595 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:22.483464 22595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:22.488567 22595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 19:20:22.491393 22619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:22.494033 22595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:20:22.494197 22595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "b281af738432475bbde220721cb47252"
format_stamp: "Formatted at 2026-08-10 19:20:19 on dist-test-slave-7lbf"
server_key: "dd090d973bd0dd1fcc43da33e6d784ba"
server_key_iv: "dfe689bc9db253e30964d6eb7ea44b7b"
server_key_version: "encryptionkey@0"
I20260810 19:20:22.495206 22595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:22.509406 22595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:22.514081 22595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:22.514587 22595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:22.515668 22595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:22.516983 22595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:22.517079 22595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:22.517189 22595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:22.517262 22595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:22.548835 22595 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:37555
I20260810 19:20:22.548895 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:37555 every 8 connection(s)
I20260810 19:20:22.550361 22595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:22.555392 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 22595
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 19:20:22.567402 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.552837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:51515 (local address 127.7.43.62:43247)
0810 19:20:22.552998 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.553003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.554240 (+ 1237us) server_negotiation.cc:407] Connection header received
0810 19:20:22.556364 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.556370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.556451 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.556553 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.558888 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.559820 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.561495 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.562034 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.562153 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.567159 (+ 5006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:51515: BlockingRecv error: recv got EOF from 127.7.43.3:51515 (error 108)
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":49}
I20260810 19:20:22.568248 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.553292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:51515)
0810 19:20:22.554063 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.554108 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.555893 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.557030 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.557043 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.557845 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.558690 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.558717 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.559977 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.559981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.561353 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.561367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.562233 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.562238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.562433 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.563035 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.563057 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.566995 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":315,"threads_started":1}
W20260810 19:20:22.569079 22799 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:43247 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.43.62:43247: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
I20260810 19:20:22.573360 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.556398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.7.43.62:43247)
0810 19:20:22.559231 (+ 2833us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.559235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.559249 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:20:22.559289 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.559292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.559343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.559422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.561081 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.562014 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.563322 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.563936 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.564062 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.566586 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:22.566598 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:22.566620 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:22.566670 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:22.570232 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:22.570889 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:22.570896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:22.570902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:22.571067 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:22.571483 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:22.571486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:22.571487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:22.571966 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:22.572256 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:22.572619 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 19:20:22.572959 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":2747,"thread_start_us":2631,"threads_started":1}
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:22.577778 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.569633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:48657)
0810 19:20:22.569811 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.569830 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.569930 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.570238 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.570243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.570569 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.570700 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.570709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.571891 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.571896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.573224 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.573239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.574133 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.574138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.574259 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.574718 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.574734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.577477 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 19:20:22.578047 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.569743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:48657 (local address 127.7.43.62:43247)
0810 19:20:22.569894 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.569898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.569913 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:20:22.570061 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.570064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.570120 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.570191 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.570834 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.571762 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.573360 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.574004 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.574121 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.577718 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:48657: BlockingRecv error: recv got EOF from 127.7.43.3:48657 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":38}
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:22.585361 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.578370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:50407)
0810 19:20:22.578535 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.578553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.578635 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.579010 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.579014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.579292 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.579429 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.579437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.580368 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.580374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.581250 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.581261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.581949 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.581955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.582107 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.582580 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.582597 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.585086 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 19:20:22.585670 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.578565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:50407 (local address 127.7.43.62:43247)
0810 19:20:22.578785 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.578789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.578802 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:20:22.578836 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.578840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.578890 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.578965 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.579556 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.580236 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.581384 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.581813 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.581920 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.585294 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:50407: BlockingRecv error: recv got EOF from 127.7.43.3:50407 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260810 19:20:22.585657 22799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:22 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:22.671617 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:22.663088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:60533)
0810 19:20:22.663594 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:22.663610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:22.663727 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:22.664034 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:22.664037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:22.664508 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:22.664720 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.664730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.665743 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.665747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.666778 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:22.666792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.667543 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.667547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:22.667668 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.668206 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:22.668222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:22.671345 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
W20260810 19:20:22.671856 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:22.663130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:60533 (local address 127.7.43.62:43247)
0810 19:20:22.663300 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:22.663304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:22.663688 (+ 384us) server_negotiation.cc:407] Connection header received
0810 19:20:22.663825 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:22.663829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:22.663923 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:22.664006 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:22.664856 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.665636 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.666942 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:22.667397 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:22.667493 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:22.671625 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:60533: BlockingRecv error: recv got EOF from 127.7.43.1:60533 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:20:23 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:23 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:23.144328 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:23.135840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:35621)
0810 19:20:23.136386 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:23.136401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:23.136558 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:23.136889 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:23.136894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:23.137395 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:23.137653 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:23.137665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.138648 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.138654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:23.139640 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:23.139653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.140470 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.140477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:23.140654 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:23.141273 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:23.141290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:23.144023 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":271,"threads_started":1}
W20260810 19:20:23.144490 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:23.135882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:35621 (local address 127.7.43.62:43247)
0810 19:20:23.136104 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:23.136111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:23.136505 (+ 394us) server_negotiation.cc:407] Connection header received
0810 19:20:23.136688 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:23.136692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:23.136757 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:23.136840 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:23.137791 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.138507 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.139789 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.140270 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.140399 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:23.144253 (+ 3854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:35621: BlockingRecv error: recv got EOF from 127.7.43.2:35621 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 19:20:23 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:23 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:23.597611 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:23.586264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:43927)
0810 19:20:23.586825 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:23.586847 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:23.587000 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:23.587349 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:23.587355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:23.587853 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:23.588066 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:23.588077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.589032 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.589035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:23.590071 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:23.590084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.590734 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.590739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:23.590859 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:23.594491 (+ 3632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:23.594515 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:23.597322 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":325,"threads_started":1}
W20260810 19:20:23.597802 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:23.586408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:43927 (local address 127.7.43.62:43247)
0810 19:20:23.586579 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:23.586584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:23.586942 (+ 358us) server_negotiation.cc:407] Connection header received
0810 19:20:23.587118 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:23.587124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:23.587200 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:23.587281 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:23.588190 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.588900 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.590183 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.590583 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.590676 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:23.597547 (+ 6871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:43927: BlockingRecv error: recv got EOF from 127.7.43.3:43927 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 19:20:23 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:23 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:23.680433 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:23.672483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:37033)
0810 19:20:23.673010 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:23.673026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:23.673175 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:23.673537 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:23.673543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:23.673959 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:23.674112 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:23.674121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.675125 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.675131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:23.676039 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:23.676051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.676748 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.676754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:23.676952 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:23.677543 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:23.677560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:23.680198 (+ 2638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":294,"threads_started":1}
W20260810 19:20:23.680627 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:23.672570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:37033 (local address 127.7.43.62:43247)
0810 19:20:23.672751 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:23.672757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:23.673112 (+ 355us) server_negotiation.cc:407] Connection header received
0810 19:20:23.673293 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:23.673296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:23.673371 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:23.673459 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:23.674251 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.674989 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.676171 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:23.676594 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:23.676711 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:23.680380 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:37033: BlockingRecv error: recv got EOF from 127.7.43.1:37033 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 19:20:24 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:24 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:24.153322 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:24.145273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:58643)
0810 19:20:24.145869 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:24.145886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:24.145999 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:24.146267 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:24.146270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:24.146677 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:24.146823 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:24.146832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.147794 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.147797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:24.148815 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:24.148831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.149488 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.149492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:24.149643 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:24.150175 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:24.150193 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:24.153021 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":358,"threads_started":1}
W20260810 19:20:24.153440 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:24.145359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:58643 (local address 127.7.43.62:43247)
0810 19:20:24.145535 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:24.145541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:24.145962 (+ 421us) server_negotiation.cc:407] Connection header received
0810 19:20:24.146088 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:24.146092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:24.146155 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:24.146238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:24.146957 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.147656 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.148951 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.149359 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.149452 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:24.153211 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:58643: BlockingRecv error: recv got EOF from 127.7.43.2:58643 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 19:20:24 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:24 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:24.606398 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:24.598684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:44385)
0810 19:20:24.599204 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:24.599226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:24.599369 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:24.599729 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:24.599732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:24.600127 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:24.600325 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:24.600338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.601265 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.601268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:24.602188 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:24.602199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.602835 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.602839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:24.602943 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:24.603468 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:24.603484 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:24.606159 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
W20260810 19:20:24.606640 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:24.598840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:44385 (local address 127.7.43.62:43247)
0810 19:20:24.599006 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:24.599012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:24.599336 (+ 324us) server_negotiation.cc:407] Connection header received
0810 19:20:24.599529 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:24.599535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:24.599611 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:24.599691 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:24.600453 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.601155 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.602299 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.602722 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.602814 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:24.606384 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:44385: BlockingRecv error: recv got EOF from 127.7.43.3:44385 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 19:20:24 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:24 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:24.688905 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:24.681350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:38105)
0810 19:20:24.681885 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:24.681901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:24.682031 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:24.682362 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:24.682366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:24.682715 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:24.682882 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:24.682890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.683851 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.683854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:24.684760 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:24.684771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.685352 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.685357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:24.685506 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:24.686081 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:24.686098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:24.688656 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":298,"threads_started":1}
W20260810 19:20:24.689046 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:24.681433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:38105 (local address 127.7.43.62:43247)
0810 19:20:24.681667 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:24.681673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:24.681982 (+ 309us) server_negotiation.cc:407] Connection header received
0810 19:20:24.682139 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:24.682145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:24.682223 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:24.682319 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:24.682995 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.683733 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.684863 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:24.685242 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:24.685327 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:24.688840 (+ 3513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:38105: BlockingRecv error: recv got EOF from 127.7.43.1:38105 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 19:20:25 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:25 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:25.164904 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:25.154327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:44737)
0810 19:20:25.154843 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:25.154861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:25.155002 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:25.155369 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:25.155374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:25.155823 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:25.156049 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:25.156059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.157531 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.157538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:25.159182 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:25.159199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.160151 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.160157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:25.160332 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:25.160972 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:25.160994 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:25.164527 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":269,"threads_started":1}
W20260810 19:20:25.165215 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:25.154393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:44737 (local address 127.7.43.62:43247)
0810 19:20:25.154589 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:25.154596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:25.154953 (+ 357us) server_negotiation.cc:407] Connection header received
0810 19:20:25.155119 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:25.155125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:25.155212 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:25.155327 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:25.156212 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.157385 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.159325 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.160001 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.160132 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:25.164738 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:44737: BlockingRecv error: recv got EOF from 127.7.43.2:44737 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 19:20:25 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:25 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:25.615895 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:25.607331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:48217)
0810 19:20:25.608032 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:25.608052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:25.608193 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:25.608558 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:25.608562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:25.609014 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:25.609212 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:25.609223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.610270 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.610276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:25.611317 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:25.611331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.612083 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.612097 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:25.612268 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:25.612865 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:25.612883 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:25.615553 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":446,"threads_started":1}
W20260810 19:20:25.616009 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:25.607419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:48217 (local address 127.7.43.62:43247)
0810 19:20:25.607653 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:25.607660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:25.608136 (+ 476us) server_negotiation.cc:407] Connection header received
0810 19:20:25.608325 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:25.608330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:25.608412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:25.608497 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:25.609354 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.610134 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.611471 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.611929 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.612044 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:25.615769 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:48217: BlockingRecv error: recv got EOF from 127.7.43.3:48217 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 19:20:25 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:25 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:25.697829 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:25.689879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:57841)
0810 19:20:25.690357 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:25.690374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:25.690515 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:25.690955 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:25.690958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:25.691445 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:25.691652 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:25.691666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.692671 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.692675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:25.693674 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:25.693687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.694322 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.694326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:25.694455 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:25.694989 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:25.695005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:25.697599 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
W20260810 19:20:25.697995 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:25.689992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:57841 (local address 127.7.43.62:43247)
0810 19:20:25.690213 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:25.690219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:25.690462 (+ 243us) server_negotiation.cc:407] Connection header received
0810 19:20:25.690657 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:25.690663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:25.690742 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:25.690920 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:25.691789 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.692569 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.693794 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:25.694216 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:25.694308 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:25.697778 (+ 3470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:57841: BlockingRecv error: recv got EOF from 127.7.43.1:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 19:20:26 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:26 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:26.178507 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:26.170351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:48183)
0810 19:20:26.170790 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:26.170804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:26.170915 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:26.171195 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:26.171198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:26.171565 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:26.171726 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:26.171735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.172659 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.172664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:26.173836 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:26.173853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.174610 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.174616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:26.174800 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:26.175421 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:26.175445 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:26.178279 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":241,"threads_started":1}
W20260810 19:20:26.178647 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:26.170424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:48183 (local address 127.7.43.62:43247)
0810 19:20:26.170664 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:26.170670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:26.170872 (+ 202us) server_negotiation.cc:407] Connection header received
0810 19:20:26.171027 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:26.171030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:26.171092 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:26.171167 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:26.171827 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.172519 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.173975 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.174458 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.174579 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:26.178453 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:48183: BlockingRecv error: recv got EOF from 127.7.43.2:48183 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 19:20:26 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:26 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:26.625025 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:26.616885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:34811)
0810 19:20:26.617353 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:26.617370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:26.617483 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:26.617797 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:26.617800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:26.618182 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:26.618339 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:26.618349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.619292 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.619295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:26.620268 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:26.620279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.620996 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.621002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:26.621199 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:26.621905 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:26.621929 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:26.624728 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":258,"threads_started":1}
W20260810 19:20:26.625120 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:26.616975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:34811 (local address 127.7.43.62:43247)
0810 19:20:26.617131 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:26.617136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:26.617443 (+ 307us) server_negotiation.cc:407] Connection header received
0810 19:20:26.617608 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:26.617612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:26.617687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:26.617766 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:26.618458 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.619193 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.620385 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.620850 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.620963 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:26.624899 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:34811: BlockingRecv error: recv got EOF from 127.7.43.3:34811 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 19:20:26 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:26 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:26.706679 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:26.698818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:40575)
0810 19:20:26.699378 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:26.699395 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:26.699509 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:26.699774 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:26.699777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:26.700200 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:26.700364 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:26.700372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.701300 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.701303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:26.702277 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:26.702291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.702940 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.702944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:26.703065 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:26.703589 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:26.703605 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:26.706441 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":266,"threads_started":1}
W20260810 19:20:26.706815 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:26.698902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:40575 (local address 127.7.43.62:43247)
0810 19:20:26.699065 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:26.699069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:26.699472 (+ 403us) server_negotiation.cc:407] Connection header received
0810 19:20:26.699609 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:26.699612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:26.699675 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:26.699743 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:26.700502 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.701204 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.702406 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:26.702821 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:26.702913 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:26.706627 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:40575: BlockingRecv error: recv got EOF from 127.7.43.1:40575 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 19:20:27 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:27 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:27.187477 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:27.179386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:46971)
0810 19:20:27.179874 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:27.179890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:27.180012 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:27.180275 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:27.180278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:27.180656 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:27.180827 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:27.180836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.181896 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.181901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:27.182844 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:27.182857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.183593 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.183599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:27.183760 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:27.184309 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:27.184328 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:27.187137 (+ 2809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":239,"threads_started":1}
W20260810 19:20:27.187608 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:27.179559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:46971 (local address 127.7.43.62:43247)
0810 19:20:27.179770 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:27.179777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:27.179979 (+ 202us) server_negotiation.cc:407] Connection header received
0810 19:20:27.180106 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:27.180109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:27.180172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:27.180248 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:27.180936 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.181747 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.182993 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.183442 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.183554 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:27.187365 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:46971: BlockingRecv error: recv got EOF from 127.7.43.2:46971 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 19:20:27 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:27 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:27.634404 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:27.626118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:43687)
0810 19:20:27.626680 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:27.626700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:27.626841 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:27.627251 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:27.627255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:27.627687 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:27.627858 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:27.627868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.628840 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.628843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:27.629800 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:27.629812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.630500 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.630504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:27.630653 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:27.631255 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:27.631280 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:27.634162 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
W20260810 19:20:27.634573 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:27.626223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:43687 (local address 127.7.43.62:43247)
0810 19:20:27.626419 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:27.626426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:27.626805 (+ 379us) server_negotiation.cc:407] Connection header received
0810 19:20:27.627031 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:27.627035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:27.627119 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:27.627202 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:27.628000 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.628733 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.629933 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.630377 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.630471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:27.634354 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:43687: BlockingRecv error: recv got EOF from 127.7.43.3:43687 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 19:20:27 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:27 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:27.715551 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:27.707601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:34093)
0810 19:20:27.708106 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:27.708122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:27.708237 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:27.708578 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:27.708582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:27.708984 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:27.709142 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:27.709151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.710247 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.710252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:27.711158 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:27.711171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.711866 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.711871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:27.712037 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:27.712580 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:27.712598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:27.715243 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
W20260810 19:20:27.715708 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:27.707673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:34093 (local address 127.7.43.62:43247)
0810 19:20:27.707883 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:27.707890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:27.708201 (+ 311us) server_negotiation.cc:407] Connection header received
0810 19:20:27.708343 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:27.708349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:27.708425 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:27.708521 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:27.709279 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.710116 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.711294 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:27.711719 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:27.711825 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:27.715464 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:34093: BlockingRecv error: recv got EOF from 127.7.43.1:34093 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 19:20:28 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:28 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:28.196538 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:28.188507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:41637)
0810 19:20:28.189094 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:28.189113 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:28.189233 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:28.189540 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:28.189543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:28.189939 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:28.190096 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:28.190105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.191080 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.191084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:28.191992 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:28.192004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.192624 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.192628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:28.192749 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:28.193328 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:28.193346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:28.196228 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":335,"threads_started":1}
W20260810 19:20:28.196717 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:28.188620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:41637 (local address 127.7.43.62:43247)
0810 19:20:28.188784 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:28.188789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:28.189181 (+ 392us) server_negotiation.cc:407] Connection header received
0810 19:20:28.189320 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:28.189324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:28.189387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:28.189486 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:28.190227 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.190972 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.192101 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.192506 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.192595 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:28.196470 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:41637: BlockingRecv error: recv got EOF from 127.7.43.2:41637 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 19:20:28 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:28 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:28.647245 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:28.635490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:44411)
0810 19:20:28.636030 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:28.636050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:28.636208 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:28.636623 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:28.636628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:28.637154 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:28.637387 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:28.637400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.638452 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.638456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:28.639382 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:28.639395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.640058 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.640062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:28.640179 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:28.643994 (+ 3815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:28.644015 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:28.646978 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":297,"threads_started":1}
W20260810 19:20:28.647486 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:28.635617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:44411 (local address 127.7.43.62:43247)
0810 19:20:28.635824 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:28.635830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:28.636153 (+ 323us) server_negotiation.cc:407] Connection header received
0810 19:20:28.636331 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:28.636336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:28.636460 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:28.636569 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:28.637550 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.638326 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.639506 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.639940 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.640034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:28.647209 (+ 7175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:44411: BlockingRecv error: recv got EOF from 127.7.43.3:44411 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 19:20:28 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:28 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:28.724535 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:28.716434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:48491)
0810 19:20:28.717041 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:28.717056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:28.717189 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:28.717490 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:28.717495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:28.717956 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:28.718133 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:28.718142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.719130 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.719134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:28.720120 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:28.720133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.720896 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.720902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:28.721090 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:28.721634 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:28.721658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:28.724257 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":230,"threads_started":1}
W20260810 19:20:28.724645 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:28.716523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:48491 (local address 127.7.43.62:43247)
0810 19:20:28.716682 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:28.716688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:28.717140 (+ 452us) server_negotiation.cc:407] Connection header received
0810 19:20:28.717300 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:28.717304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:28.717365 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:28.717455 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:28.718271 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.718993 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.720248 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:28.720743 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:28.720846 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:28.724465 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:48491: BlockingRecv error: recv got EOF from 127.7.43.1:48491 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 19:20:29 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:29 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:29.206043 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:29.197462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:56957)
0810 19:20:29.198123 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:29.198139 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:29.198265 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:29.198614 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:29.198619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:29.199191 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:29.199369 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:29.199379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.200484 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.200490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:29.201402 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:29.201414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.202126 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.202132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:29.202296 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:29.202820 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:29.202837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:29.205749 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":253,"threads_started":1}
W20260810 19:20:29.206195 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:29.197583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:56957 (local address 127.7.43.62:43247)
0810 19:20:29.197779 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:29.197786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:29.198214 (+ 428us) server_negotiation.cc:407] Connection header received
0810 19:20:29.198402 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:29.198405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:29.198484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:29.198565 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:29.199520 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.200348 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.201561 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.201979 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.202087 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:29.205922 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:56957: BlockingRecv error: recv got EOF from 127.7.43.2:56957 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 19:20:29 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:29 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:29.657651 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:29.648316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:46361)
0810 19:20:29.649054 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:29.649072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:29.649205 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:29.649643 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:29.649646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:29.650135 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:29.650333 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:29.650345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.651594 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.651599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:29.652966 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:29.652982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.653796 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.653803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:29.653938 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:29.654522 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:29.654541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:29.657347 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":304,"threads_started":1}
W20260810 19:20:29.657966 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:29.648475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:46361 (local address 127.7.43.62:43247)
0810 19:20:29.648647 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:29.648652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:29.649171 (+ 519us) server_negotiation.cc:407] Connection header received
0810 19:20:29.649301 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:29.649305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:29.649465 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:29.649608 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:29.650492 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.651442 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.653105 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.653629 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.653732 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:29.657637 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:46361: BlockingRecv error: recv got EOF from 127.7.43.3:46361 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 19:20:29 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:29 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:29.735342 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:29.725763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:51485)
0810 19:20:29.726378 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:29.726401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:29.726585 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:29.727049 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:29.727055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:29.727681 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:29.727859 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:29.727872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.728979 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.728983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:29.730000 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:29.730019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.731084 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.731089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:29.731223 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:29.731868 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:29.731886 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:29.734972 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
W20260810 19:20:29.735507 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:29.725894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:51485 (local address 127.7.43.62:43247)
0810 19:20:29.726128 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:29.726135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:29.726491 (+ 356us) server_negotiation.cc:407] Connection header received
0810 19:20:29.726754 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:29.726760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:29.726841 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:29.726932 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:29.728073 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.728844 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.730158 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:29.730903 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:29.731006 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:29.735231 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:51485: BlockingRecv error: recv got EOF from 127.7.43.1:51485 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 19:20:30 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:30 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:30.215266 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:30.206991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:60041)
0810 19:20:30.207503 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:30.207520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:30.207636 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:30.207912 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:30.207914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:30.208318 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:30.208491 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:30.208499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.209753 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.209757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:30.210679 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:30.210691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.211466 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.211473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:30.211620 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:30.212284 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:30.212302 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:30.214957 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
W20260810 19:20:30.215420 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:30.207111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:60041 (local address 127.7.43.62:43247)
0810 19:20:30.207328 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:30.207334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:30.207591 (+ 257us) server_negotiation.cc:407] Connection header received
0810 19:20:30.207733 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:30.207737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:30.207802 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:30.207883 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:30.208652 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.209610 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.210819 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.211278 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.211388 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:30.215210 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:60041: BlockingRecv error: recv got EOF from 127.7.43.2:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 19:20:30 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:30 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:30.666770 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:30.658660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:34499)
0810 19:20:30.659188 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:30.659206 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:30.659323 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:30.659629 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:30.659632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:30.660068 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:30.660257 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:30.660270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.661412 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.661418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:30.662586 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:30.662599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.663291 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.663295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:30.663415 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:30.663990 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:30.664006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:30.666548 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
W20260810 19:20:30.666983 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:30.658756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:34499 (local address 127.7.43.62:43247)
0810 19:20:30.658957 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:30.658962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:30.659278 (+ 316us) server_negotiation.cc:407] Connection header received
0810 19:20:30.659413 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:30.659417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:30.659510 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:30.659596 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:30.660399 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.661258 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.662721 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.663142 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.663240 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:30.666742 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:34499: BlockingRecv error: recv got EOF from 127.7.43.3:34499 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 19:20:30 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:30 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:30.744969 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:30.736230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:44473)
0810 19:20:30.736764 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:30.736780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:30.736896 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:30.737180 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:30.737183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:30.737621 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:30.737807 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:30.737816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.738906 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.738910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:30.739840 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:30.739854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.740641 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.740647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:30.740837 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:30.741389 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:30.741407 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:30.744675 (+ 3268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":297,"threads_started":1}
W20260810 19:20:30.745333 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:30.736289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:44473 (local address 127.7.43.62:43247)
0810 19:20:30.736462 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:30.736467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:30.736844 (+ 377us) server_negotiation.cc:407] Connection header received
0810 19:20:30.737006 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:30.737009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:30.737070 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:30.737148 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:30.737996 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.738753 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.739986 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:30.740483 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:30.740632 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:30.744954 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:44473: BlockingRecv error: recv got EOF from 127.7.43.1:44473 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 19:20:31 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:31 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:31.224718 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:31.216185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:44443)
0810 19:20:31.216693 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:31.216709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:31.216822 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:31.217092 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:31.217095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:31.217539 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:31.217732 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:31.217742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.218734 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.218738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:31.219879 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:31.219894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.220592 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.220596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:31.220722 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:31.221298 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:31.221325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:31.224357 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
W20260810 19:20:31.224819 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:31.216241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:44443 (local address 127.7.43.62:43247)
0810 19:20:31.216473 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:31.216481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:31.216791 (+ 310us) server_negotiation.cc:407] Connection header received
0810 19:20:31.216912 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:31.216916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:31.216981 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:31.217059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:31.217861 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.218623 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.220005 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.220453 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.220581 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:31.224580 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:44443: BlockingRecv error: recv got EOF from 127.7.43.2:44443 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 19:20:31 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:31 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:31.675515 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:31.667672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:37605)
0810 19:20:31.668183 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:31.668199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:31.668313 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:31.668622 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:31.668625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:31.669025 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:31.669202 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:31.669211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.670160 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.670164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:31.671124 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:31.671137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.671758 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.671762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:31.671878 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:31.672411 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:31.672428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:31.675259 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
W20260810 19:20:31.675755 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:31.667734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:37605 (local address 127.7.43.62:43247)
0810 19:20:31.667899 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:31.667903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:31.668266 (+ 363us) server_negotiation.cc:407] Connection header received
0810 19:20:31.668420 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:31.668423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:31.668510 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:31.668587 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:31.669313 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.670057 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.671233 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.671639 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.671728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:31.675480 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:37605: BlockingRecv error: recv got EOF from 127.7.43.3:37605 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 19:20:31 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:31 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:31.754995 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:31.746004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:55825)
0810 19:20:31.746530 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:31.746547 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:31.746684 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:31.747062 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:31.747067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:31.747483 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:31.747638 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:31.747648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.748742 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.748746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:31.749915 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:31.749930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.750679 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.750685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:31.750872 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:31.751553 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:31.751573 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:31.754694 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":314,"threads_started":1}
W20260810 19:20:31.755100 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:31.746098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:55825 (local address 127.7.43.62:43247)
0810 19:20:31.746290 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:31.746296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:31.746636 (+ 340us) server_negotiation.cc:407] Connection header received
0810 19:20:31.746841 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:31.746846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:31.746915 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:31.747005 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:31.747792 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.748602 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.750056 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:31.750518 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:31.750643 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:31.754899 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:55825: BlockingRecv error: recv got EOF from 127.7.43.1:55825 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 19:20:32 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:32 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:32.234102 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:32.225856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:48087)
0810 19:20:32.226356 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:32.226372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:32.226505 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:32.226820 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:32.226823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:32.227197 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:32.227354 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:32.227363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.228331 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.228334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:32.229270 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:32.229282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.229924 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.229929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:32.230060 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:32.230620 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:32.230636 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:32.233840 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
W20260810 19:20:32.234261 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:32.225917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:48087 (local address 127.7.43.62:43247)
0810 19:20:32.226124 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:32.226132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:32.226447 (+ 315us) server_negotiation.cc:407] Connection header received
0810 19:20:32.226607 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:32.226611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:32.226673 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:32.226767 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:32.227474 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.228225 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.229380 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.229802 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.229892 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:32.234058 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:48087: BlockingRecv error: recv got EOF from 127.7.43.2:48087 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:20:32 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:32 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:32.685118 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:32.676440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:51973)
0810 19:20:32.676975 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:32.676997 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:32.677133 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:32.677451 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:32.677456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:32.677897 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:32.678059 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:32.678072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.679106 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.679112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:32.680058 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:32.680070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.680788 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.680794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:32.680982 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:32.681574 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:32.681590 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:32.684891 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
W20260810 19:20:32.685329 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:32.676652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:51973 (local address 127.7.43.62:43247)
0810 19:20:32.676911 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:32.676917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:32.677077 (+ 160us) server_negotiation.cc:407] Connection header received
0810 19:20:32.677255 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:32.677259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:32.677319 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:32.677393 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:32.678206 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.678972 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.680202 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.680643 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.680750 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:32.685074 (+ 4324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:51973: BlockingRecv error: recv got EOF from 127.7.43.3:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 10 19:20:32 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:32 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:32.765249 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:32.756099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:53801)
0810 19:20:32.756646 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:32.756667 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:32.756814 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:32.757180 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:32.757185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:32.757715 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:32.757897 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:32.757907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.759187 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.759195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:32.760197 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:32.760214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.761052 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.761060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:32.761256 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:32.761936 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:32.761956 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:32.764877 (+ 2921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
W20260810 19:20:32.765455 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:32.756182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:53801 (local address 127.7.43.62:43247)
0810 19:20:32.756420 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:32.756427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:32.756753 (+ 326us) server_negotiation.cc:407] Connection header received
0810 19:20:32.756935 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:32.756940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:32.757035 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:32.757122 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:32.758044 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.759011 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.760366 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:32.760882 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:32.761010 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:32.765188 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:53801: BlockingRecv error: recv got EOF from 127.7.43.1:53801 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 19:20:33 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:33 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:33.243489 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:33.235034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:51271)
0810 19:20:33.235559 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:33.235574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:33.235707 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:33.236047 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:33.236051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:33.236551 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:33.236767 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:33.236781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.238002 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.238007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:33.238945 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:33.238958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.239673 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.239678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:33.239802 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:33.240342 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:33.240359 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:33.243241 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
W20260810 19:20:33.243654 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:33.235189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:51271 (local address 127.7.43.62:43247)
0810 19:20:33.235397 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:33.235403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:33.235655 (+ 252us) server_negotiation.cc:407] Connection header received
0810 19:20:33.235828 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:33.235832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:33.235911 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:33.236011 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:33.236918 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.237862 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.239080 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.239515 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.239609 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:33.243449 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:51271: BlockingRecv error: recv got EOF from 127.7.43.2:51271 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 19:20:33 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:33 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:33.697558 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:33.686063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:60685)
0810 19:20:33.686555 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:33.686572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:33.686688 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:33.686962 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:33.686966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:33.687429 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:33.687638 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:33.687646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.688615 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.688619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:33.689570 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:33.689585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.690156 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.690159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:33.690276 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:33.694360 (+ 4084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:33.694380 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:33.697201 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
W20260810 19:20:33.697705 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:33.686188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:60685 (local address 127.7.43.62:43247)
0810 19:20:33.686411 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:33.686418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:33.686646 (+ 228us) server_negotiation.cc:407] Connection header received
0810 19:20:33.686779 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:33.686783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:33.686846 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:33.686925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:33.687755 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.688484 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.689679 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.690042 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.690128 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:33.697438 (+ 7310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:60685: BlockingRecv error: recv got EOF from 127.7.43.3:60685 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 19:20:33 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:33 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:33.774354 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:33.766186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:60847)
0810 19:20:33.766723 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:33.766740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:33.766855 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:33.767179 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:33.767182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:33.767619 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:33.767793 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:33.767801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.768786 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.768789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:33.769784 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:33.769798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.770512 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.770516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:33.770655 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:33.771218 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:33.771235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:33.774030 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
W20260810 19:20:33.774499 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:33.766225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:60847 (local address 127.7.43.62:43247)
0810 19:20:33.766387 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:33.766391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:33.766805 (+ 414us) server_negotiation.cc:407] Connection header received
0810 19:20:33.766970 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:33.766973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:33.767037 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:33.767116 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:33.767928 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.768671 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.769896 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:33.770381 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:33.770487 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:33.774270 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:60847: BlockingRecv error: recv got EOF from 127.7.43.1:60847 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 19:20:34 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:34 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:34.252758 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:34.244557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:42759)
0810 19:20:34.245114 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:34.245130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:34.245269 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:34.245679 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:34.245684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:34.246123 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:34.246284 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:34.246293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.247311 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.247316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:34.248263 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:34.248275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.248964 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.248970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:34.249145 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:34.249725 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:34.249743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:34.252457 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":292,"threads_started":1}
W20260810 19:20:34.252892 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:34.244589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:42759 (local address 127.7.43.62:43247)
0810 19:20:34.244792 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:34.244799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:34.245191 (+ 392us) server_negotiation.cc:407] Connection header received
0810 19:20:34.245379 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:34.245384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:34.245491 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:34.245613 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:34.246434 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.247169 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.248408 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.248818 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.248928 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:34.252677 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:42759: BlockingRecv error: recv got EOF from 127.7.43.2:42759 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 19:20:34 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:34 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:34.710397 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:34.698595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:38185)
0810 19:20:34.699195 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:34.699211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:34.699378 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:34.699827 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:34.699832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:34.700336 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:34.700529 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:34.700540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.701674 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.701678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:34.702616 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:34.702629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.703306 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.703310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:34.703455 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:34.707145 (+ 3690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:34.707166 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:34.710062 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":347,"threads_started":1}
W20260810 19:20:34.710631 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:34.698694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:38185 (local address 127.7.43.62:43247)
0810 19:20:34.698974 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:34.698981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:34.699331 (+ 350us) server_negotiation.cc:407] Connection header received
0810 19:20:34.699531 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:34.699538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:34.699661 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:34.699779 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:34.700659 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.701538 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.702746 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.703187 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.703286 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:34.710347 (+ 7061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:38185: BlockingRecv error: recv got EOF from 127.7.43.3:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 19:20:34 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:34 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:34.783387 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:34.775300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:42303)
0810 19:20:34.775782 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:34.775799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:34.775911 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:34.776195 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:34.776198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:34.776601 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:34.776761 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:34.776771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.777791 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.777796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:34.778834 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:34.778849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.779528 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.779533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:34.779648 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:34.780263 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:34.780281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:34.783082 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":248,"threads_started":1}
W20260810 19:20:34.783545 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:34.775392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:42303 (local address 127.7.43.62:43247)
0810 19:20:34.775623 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:34.775629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:34.775870 (+ 241us) server_negotiation.cc:407] Connection header received
0810 19:20:34.776015 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:34.776019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:34.776083 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:34.776160 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:34.776881 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.777678 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.778962 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:34.779375 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:34.779471 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:34.783284 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:42303: BlockingRecv error: recv got EOF from 127.7.43.1:42303 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 19:20:35 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:35 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:35.261598 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:35.253750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:56975)
0810 19:20:35.254239 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:35.254255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:35.254364 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:35.254648 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:35.254651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:35.255043 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:35.255229 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:35.255238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.256206 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.256209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:35.257136 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:35.257147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.257825 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.257829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:35.257945 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:35.258461 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:35.258477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:35.261266 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":240,"threads_started":1}
W20260810 19:20:35.261758 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:35.253906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:56975 (local address 127.7.43.62:43247)
0810 19:20:35.254121 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:35.254127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:35.254318 (+ 191us) server_negotiation.cc:407] Connection header received
0810 19:20:35.254463 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:35.254466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:35.254532 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:35.254611 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:35.255336 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.256104 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.257246 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.257692 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.257778 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:35.261533 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:56975: BlockingRecv error: recv got EOF from 127.7.43.2:56975 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 19:20:35 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:35 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:35.719796 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:35.711332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:41971)
0810 19:20:35.711879 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:35.711896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:35.712023 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:35.712393 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:35.712397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:35.712878 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:35.713082 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:35.713092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.714185 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.714190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:35.715149 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:35.715161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.715845 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.715849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:35.715981 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:35.716517 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:35.716533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:35.719490 (+ 2957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":287,"threads_started":1}
W20260810 19:20:35.719970 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:35.711445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:41971 (local address 127.7.43.62:43247)
0810 19:20:35.711638 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:35.711643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:35.711974 (+ 331us) server_negotiation.cc:407] Connection header received
0810 19:20:35.712113 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:35.712116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:35.712179 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:35.712260 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:35.713266 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.714052 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.715274 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.715704 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.715797 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:35.719681 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:41971: BlockingRecv error: recv got EOF from 127.7.43.3:41971 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 19:20:35 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:35 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:35.793027 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:35.784376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:34421)
0810 19:20:35.784875 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:35.784891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:35.785016 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:35.785404 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:35.785406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:35.785864 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:35.786080 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:35.786090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.787097 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.787100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:35.788024 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:35.788036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.788721 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.788725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:35.788847 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:35.789467 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:35.789489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:35.792671 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
W20260810 19:20:35.793136 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:35.784480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:34421 (local address 127.7.43.62:43247)
0810 19:20:35.784746 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:35.784751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:35.784971 (+ 220us) server_negotiation.cc:407] Connection header received
0810 19:20:35.785184 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:35.785189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:35.785264 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:35.785376 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:35.786220 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.786975 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.788168 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:35.788608 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:35.788705 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:35.792895 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:34421: BlockingRecv error: recv got EOF from 127.7.43.1:34421 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Aug 10 19:20:36 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:36 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:36.271258 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:36.262495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:60639)
0810 19:20:36.263030 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:36.263047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:36.263177 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:36.263504 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:36.263509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:36.263950 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:36.264110 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:36.264120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.265168 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.265174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:36.266133 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:36.266146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.266854 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.266860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:36.267034 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:36.267605 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:36.267622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:36.270929 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":272,"threads_started":1}
W20260810 19:20:36.271355 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:36.262630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:60639 (local address 127.7.43.62:43247)
0810 19:20:36.262822 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:36.262828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:36.263130 (+ 302us) server_negotiation.cc:407] Connection header received
0810 19:20:36.263301 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:36.263304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:36.263366 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:36.263468 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:36.264258 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.265023 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.266268 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.266710 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.266820 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:36.271134 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:60639: BlockingRecv error: recv got EOF from 127.7.43.2:60639 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 19:20:36 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:36 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.3@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:36.729432 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:36.720943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.3:54615)
0810 19:20:36.721479 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:36.721500 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:36.721690 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:36.722103 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:36.722108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:36.722569 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:36.722769 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:36.722780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.723816 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.723822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:36.724748 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:36.724759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.725487 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.725493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:36.725683 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:36.726264 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:36.726280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:36.729097 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":274,"threads_started":1}
W20260810 19:20:36.729609 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:36.721001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:54615 (local address 127.7.43.62:43247)
0810 19:20:36.721180 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:36.721185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:36.721618 (+ 433us) server_negotiation.cc:407] Connection header received
0810 19:20:36.721822 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:36.721827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:36.721911 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:36.722038 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:36.722907 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.723670 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.724881 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.725342 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.725450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:36.729357 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.3:54615: BlockingRecv error: recv got EOF from 127.7.43.3:54615 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 19:20:36 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:36 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.1@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:36.802335 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:36.794221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.1:57291)
0810 19:20:36.794735 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:36.794751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:36.794859 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:36.795128 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:36.795131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:36.795533 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:36.795725 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:36.795734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.796660 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.796663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:36.797666 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:36.797679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.798425 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.798431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:36.798552 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:36.799164 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:36.799182 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:36.801989 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":270,"threads_started":1}
W20260810 19:20:36.802453 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:36.794293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:57291 (local address 127.7.43.62:43247)
0810 19:20:36.794466 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:36.794472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:36.794821 (+ 349us) server_negotiation.cc:407] Connection header received
0810 19:20:36.794977 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:36.794981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:36.795045 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:36.795114 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:36.795851 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.796554 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.797796 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:36.798291 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:36.798389 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:36.802221 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.1:57291: BlockingRecv error: recv got EOF from 127.7.43.1:57291 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 19:20:37 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 19:20:37 dist-test-slave-7lbf krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.43.2@KRBTEST.COM for oryx/127.7.43.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 19:20:37.280230 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:37.272398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43247 (local address 127.7.43.2:48555)
0810 19:20:37.272869 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:37.272885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:37.273014 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:37.273279 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:37.273283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:37.273745 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:37.273947 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:37.273957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.274898 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.274902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:37.275812 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:37.275823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.276454 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.276458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:37.276564 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:37.277130 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:37.277148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:37.279864 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.43.62:43247: Server oryx/127.7.43.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
W20260810 19:20:37.280336 22364 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:37.272511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:48555 (local address 127.7.43.62:43247)
0810 19:20:37.272722 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:37.272729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:37.272964 (+ 235us) server_negotiation.cc:407] Connection header received
0810 19:20:37.273119 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:37.273123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:37.273184 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:37.273251 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:37.274057 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.274794 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.275913 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.276352 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.276439 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:37.280086 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.43.2:48555: BlockingRecv error: recv got EOF from 127.7.43.2:48555 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:20:37.566391 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22105
I20260810 19:20:37.578229 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:37.567464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.7.43.1:35235)
0810 19:20:37.567985 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:37.567993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:37.568019 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:20:37.568088 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:37.568096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:37.568307 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:37.568475 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:37.569874 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.570870 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.571977 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.572518 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.572608 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:37.575010 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:37.575019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:37.575033 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:37.575069 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:37.577718 (+ 2649us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.43.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 19:20:37.577965 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36056: Request ticket server kudu/127.7.43.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
W20260810 19:20:37.578505 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22105: Not authorized: Client connection negotiation failed: client connection to 127.7.43.1:35235: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.43.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 19:20:37.578655 22869 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36056: Request ticket server kudu/127.7.43.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 19:20:37.578716 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 22105
I20260810 19:20:37.608409 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22370
I20260810 19:20:37.620908 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:37.609602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.7.43.2:46107)
0810 19:20:37.610274 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:37.610282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:37.610313 (+ 31us) server_negotiation.cc:407] Connection header received
0810 19:20:37.610375 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:37.610381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:37.610636 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:37.610827 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:37.612152 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.613251 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.614547 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.615129 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.615248 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:37.617772 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:37.617781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:37.617797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:37.617835 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:37.620196 (+ 2361us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.43.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 19:20:37.620578 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47134: Request ticket server kudu/127.7.43.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
W20260810 19:20:37.620893 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22370: Not authorized: Client connection negotiation failed: client connection to 127.7.43.2:46107: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.43.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 19:20:37.621004 22871 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47134: Request ticket server kudu/127.7.43.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 19:20:37.621099 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 22370
I20260810 19:20:37.645366 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22595
I20260810 19:20:37.658568 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:37.646534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.7.43.3:37555)
0810 19:20:37.647329 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:37.647336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:37.647363 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:20:37.647426 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:37.647433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:37.647650 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:37.647822 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:37.649247 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.650553 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.651773 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:37.652338 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:37.652455 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:37.655133 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:37.655141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:37.655156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:37.655192 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:37.657904 (+ 2712us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.43.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 19:20:37.658231 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39648: Request ticket server kudu/127.7.43.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":491,"threads_started":1}
W20260810 19:20:37.658687 22873 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39648: Request ticket server kudu/127.7.43.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 19:20:37.658797 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22595: Not authorized: Client connection negotiation failed: client connection to 127.7.43.3:37555: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.43.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 19:20:37.658974 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 22595
I20260810 19:20:37.682884 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 21985
I20260810 19:20:37.684386 22092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:37.802242 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 21985
2026-08-10T19:20:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21416 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T19:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:37Z Disabled control of system clock
I20260810 19:20:37.899655 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:45021
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 19:20:38.236726 22878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:38.237037 22878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:38.237128 22878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:38.244509 22878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:38.244619 22878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:38.244656 22878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:38.244712 22878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:38.244756 22878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:38.254196 22878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:45021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:38.255538 22878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:38.257390 22878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:38.269505 22886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:38.269462 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:38.269543 22891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:38.269878 22878 server_base.cc:1034] running on GCE node
I20260810 19:20:38.270959 22878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:38.272440 22878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:38.273607 22878 hybrid_clock.cc:648] HybridClock initialized: now 1786389638273557 us; error 36 us; skew 500 ppm
I20260810 19:20:38.273972 22878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:38.276288 22878 webserver.cc:460] Webserver started at http://127.7.43.62:42963/ using document root <none> and password file <none>
I20260810 19:20:38.277012 22878 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:38.277115 22878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:38.277454 22878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:38.280452 22878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "06aafb4d7512446a8fc89dd7e129674b"
format_stamp: "Formatted at 2026-08-10 19:20:38 on dist-test-slave-7lbf"
server_key: "d1c416f8cba2cea32bf3fc1b54a6d746"
server_key_iv: "c64e75a86fdb1acfd1e72a4c51fd3f8e"
server_key_version: "encryptionkey@0"
I20260810 19:20:38.281240 22878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "06aafb4d7512446a8fc89dd7e129674b"
format_stamp: "Formatted at 2026-08-10 19:20:38 on dist-test-slave-7lbf"
server_key: "d1c416f8cba2cea32bf3fc1b54a6d746"
server_key_iv: "c64e75a86fdb1acfd1e72a4c51fd3f8e"
server_key_version: "encryptionkey@0"
I20260810 19:20:38.286525 22878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:20:38.290098 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:38.291481 22878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:20:38.291644 22878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "06aafb4d7512446a8fc89dd7e129674b"
format_stamp: "Formatted at 2026-08-10 19:20:38 on dist-test-slave-7lbf"
server_key: "d1c416f8cba2cea32bf3fc1b54a6d746"
server_key_iv: "c64e75a86fdb1acfd1e72a4c51fd3f8e"
server_key_version: "encryptionkey@0"
I20260810 19:20:38.291849 22878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:38.309006 22878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:38.309902 22878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:38.310189 22878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:38.327720 22878 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:45021
I20260810 19:20:38.327744 22984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:45021 every 8 connection(s)
I20260810 19:20:38.329090 22878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:20:38.332868 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:38.335793 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 22878
I20260810 19:20:38.335999 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:20:38.336637 7340 external_mini_cluster.cc:1428] Setting key fbee3cd2e188e48901d9d6317e8cfd6c
I20260810 19:20:38.346591 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b: Bootstrap starting.
I20260810 19:20:38.350509 22986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:38.351962 22986 log.cc:824] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:38.352358 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:38.338776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.7.43.62:45021)
0810 19:20:38.339882 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:38.339898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:38.339965 (+ 67us) server_negotiation.cc:407] Connection header received
0810 19:20:38.342011 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:38.342044 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:38.342725 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:38.343211 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:38.343792 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:38.345731 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:38.347081 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:38.348105 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:38.348326 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:38.349350 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:38.349361 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:38.349379 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:38.350150 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:38.350275 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:38.350308 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:38.350454 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:38.350685 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 19:20:38.351052 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":342,"threads_started":1}
I20260810 19:20:38.356019 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b: No bootstrap required, opened a new log
I20260810 19:20:38.363304 22986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aafb4d7512446a8fc89dd7e129674b" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45021 } }
I20260810 19:20:38.363673 22986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:38.363749 22986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06aafb4d7512446a8fc89dd7e129674b, State: Initialized, Role: FOLLOWER
I20260810 19:20:38.364252 22986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aafb4d7512446a8fc89dd7e129674b" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45021 } }
I20260810 19:20:38.364465 22986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:20:38.364564 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:20:38.364684 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:38.366958 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aafb4d7512446a8fc89dd7e129674b" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45021 } }
I20260810 19:20:38.367419 22986 leader_election.cc:302] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06aafb4d7512446a8fc89dd7e129674b; no voters:
I20260810 19:20:38.367944 22986 leader_election.cc:288] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:20:38.368090 22995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:38.368711 22995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [term 1 LEADER]: Becoming Leader. State: Replica: 06aafb4d7512446a8fc89dd7e129674b, State: Running, Role: LEADER
I20260810 19:20:38.369233 22995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aafb4d7512446a8fc89dd7e129674b" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45021 } }
I20260810 19:20:38.370034 22986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:38.372010 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06aafb4d7512446a8fc89dd7e129674b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aafb4d7512446a8fc89dd7e129674b" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45021 } } }
I20260810 19:20:38.372387 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:38.374218 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06aafb4d7512446a8fc89dd7e129674b. Latest consensus state: current_term: 1 leader_uuid: "06aafb4d7512446a8fc89dd7e129674b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aafb4d7512446a8fc89dd7e129674b" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45021 } } }
I20260810 19:20:38.374539 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:38.377830 23003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:38.382216 23003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:38.390563 23003 catalog_manager.cc:1347] Generated new cluster ID: 6aed5ee1e13241049b3265980040b3cf
I20260810 19:20:38.390686 23003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:38.406656 23003 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:20:38.407914 23003 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:38.416703 23003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06aafb4d7512446a8fc89dd7e129674b: Generated new TSK 0
I20260810 19:20:38.417559 23003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:20:38.428099 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45021
--builtin_ntp_servers=127.7.43.20:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 19:20:38.768129 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:38.768450 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:38.768563 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:38.775979 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:38.776093 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:38.776242 23025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:38.785633 23025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:38.787053 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:38.788902 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:38.803440 23038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:38.803467 23033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:38.803496 23034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:38.804235 23025 server_base.cc:1034] running on GCE node
I20260810 19:20:38.804929 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:38.805933 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:38.807144 23025 hybrid_clock.cc:648] HybridClock initialized: now 1786389638807103 us; error 32 us; skew 500 ppm
I20260810 19:20:38.807475 23025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:38.812678 23025 webserver.cc:460] Webserver started at http://127.7.43.1:44237/ using document root <none> and password file <none>
I20260810 19:20:38.813403 23025 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:38.813539 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:38.813862 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:38.816749 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "99fb94c050824009b4500becc059abb0"
format_stamp: "Formatted at 2026-08-10 19:20:38 on dist-test-slave-7lbf"
server_key: "df72c36c0740ce3b5a134f568a20cb2e"
server_key_iv: "42f49a7c4184bc26ca07041a9fac24d7"
server_key_version: "encryptionkey@0"
I20260810 19:20:38.817566 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "99fb94c050824009b4500becc059abb0"
format_stamp: "Formatted at 2026-08-10 19:20:38 on dist-test-slave-7lbf"
server_key: "df72c36c0740ce3b5a134f568a20cb2e"
server_key_iv: "42f49a7c4184bc26ca07041a9fac24d7"
server_key_version: "encryptionkey@0"
I20260810 19:20:38.822755 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 19:20:38.826145 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:38.827468 23025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:38.827626 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "99fb94c050824009b4500becc059abb0"
format_stamp: "Formatted at 2026-08-10 19:20:38 on dist-test-slave-7lbf"
server_key: "df72c36c0740ce3b5a134f568a20cb2e"
server_key_iv: "42f49a7c4184bc26ca07041a9fac24d7"
server_key_version: "encryptionkey@0"
I20260810 19:20:38.827828 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:38.841267 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:38.842152 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:38.842412 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:38.843353 23025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:38.844808 23025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:38.844916 23025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:38.845031 23025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:38.845103 23025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:38.876209 23025 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34097
I20260810 19:20:38.876266 23247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34097 every 8 connection(s)
I20260810 19:20:38.877662 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:38.887568 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 23025
I20260810 19:20:38.887820 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:20:38.888675 7340 external_mini_cluster.cc:1428] Setting key f558e9462d6ae4117039657ca00ae104
I20260810 19:20:38.889998 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:38.880261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:51317 (local address 127.7.43.62:45021)
0810 19:20:38.880695 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:38.880701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:38.881983 (+ 1282us) server_negotiation.cc:407] Connection header received
0810 19:20:38.883657 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:38.883661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:38.883725 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:38.883818 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:38.884819 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:38.885573 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:38.887029 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:38.887418 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:38.887507 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:38.889111 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:38.889116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:38.889119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:38.889138 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:38.889184 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:38.889197 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:38.889266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:38.889543 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 19:20:38.889646 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":198,"threads_started":1}
I20260810 19:20:38.891084 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:38.880891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45021 (local address 127.7.43.1:51317)
0810 19:20:38.881827 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:38.881862 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:38.883419 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:38.884010 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:38.884018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:38.884106 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:38.884689 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:38.884705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:38.885686 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:38.885690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:38.886922 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:38.886934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:38.887650 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:38.887659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:38.888010 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:38.888743 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:20:38.888747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:38.888805 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:20:38.888907 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:38.889296 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:38.889307 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:38.889316 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 19:20:38.889478 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 19:20:38.889911 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":272,"threads_started":1}
I20260810 19:20:38.891969 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45021
--builtin_ntp_servers=127.7.43.20:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 19:20:38.892707 23248 heartbeater.cc:347] Connected to a master server at 127.7.43.62:45021
I20260810 19:20:38.892997 23248 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:38.893810 23248 heartbeater.cc:511] Master 127.7.43.62:45021 requested a full tablet report, sending...
I20260810 19:20:38.895867 22929 ts_manager.cc:194] Registered new tserver with Master: 99fb94c050824009b4500becc059abb0 (127.7.43.1:34097)
I20260810 19:20:38.897420 22929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.1:51317
W20260810 19:20:39.226501 23253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:39.226809 23253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:39.226892 23253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:39.234315 23253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:39.234424 23253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:39.234531 23253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:39.243646 23253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:39.244881 23253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:39.246623 23253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:39.259279 23259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:39.259241 23264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:39.259308 23260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:39.260237 23253 server_base.cc:1034] running on GCE node
I20260810 19:20:39.260953 23253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:39.261934 23253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:39.263135 23253 hybrid_clock.cc:648] HybridClock initialized: now 1786389639263085 us; error 43 us; skew 500 ppm
I20260810 19:20:39.263499 23253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:39.265833 23253 webserver.cc:460] Webserver started at http://127.7.43.2:39341/ using document root <none> and password file <none>
I20260810 19:20:39.266547 23253 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:39.266654 23253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:39.266970 23253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:39.269893 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "3cd6718a6c17468ca4e6f92ca4ab4c85"
format_stamp: "Formatted at 2026-08-10 19:20:39 on dist-test-slave-7lbf"
server_key: "c7c784d8e44bb9226aa426c046de61ef"
server_key_iv: "dad4e3ade72737a56aec2387fba398f4"
server_key_version: "encryptionkey@0"
I20260810 19:20:39.270725 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "3cd6718a6c17468ca4e6f92ca4ab4c85"
format_stamp: "Formatted at 2026-08-10 19:20:39 on dist-test-slave-7lbf"
server_key: "c7c784d8e44bb9226aa426c046de61ef"
server_key_iv: "dad4e3ade72737a56aec2387fba398f4"
server_key_version: "encryptionkey@0"
I20260810 19:20:39.276110 23253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260810 19:20:39.279590 23277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:39.280860 23253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:20:39.281009 23253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "3cd6718a6c17468ca4e6f92ca4ab4c85"
format_stamp: "Formatted at 2026-08-10 19:20:39 on dist-test-slave-7lbf"
server_key: "c7c784d8e44bb9226aa426c046de61ef"
server_key_iv: "dad4e3ade72737a56aec2387fba398f4"
server_key_version: "encryptionkey@0"
I20260810 19:20:39.281203 23253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:39.292886 23253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:39.293723 23253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:39.293973 23253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:39.294875 23253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:39.296329 23253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:39.296425 23253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:39.296535 23253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:39.296604 23253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:39.326804 23253 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:39403
I20260810 19:20:39.326858 23474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:39403 every 8 connection(s)
I20260810 19:20:39.328192 23253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:39.330101 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 23253
I20260810 19:20:39.330315 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:20:39.330868 7340 external_mini_cluster.cc:1428] Setting key ededaef2ce619308408e0cea6cf44bc5
I20260810 19:20:39.334067 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45021
--builtin_ntp_servers=127.7.43.20:44193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 19:20:39.341126 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:39.331101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:44395 (local address 127.7.43.62:45021)
0810 19:20:39.331599 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:39.331605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:39.333190 (+ 1585us) server_negotiation.cc:407] Connection header received
0810 19:20:39.335079 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:39.335084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:39.335163 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:39.335252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:39.336316 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.337061 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.338619 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.339011 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.339116 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:39.340373 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:39.340378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:39.340381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:39.340398 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:39.340429 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:39.340441 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:39.340520 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:39.340726 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:20:39.340824 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260810 19:20:39.342275 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:39.331716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45021 (local address 127.7.43.2:44395)
0810 19:20:39.333038 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:39.333091 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:39.334812 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:39.335488 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:39.335496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:39.335587 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:39.336154 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:39.336172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.337198 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.337203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:39.338484 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:39.338496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.339219 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.339227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:39.339433 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:39.340063 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:20:39.340066 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:39.340109 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:20:39.340181 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:39.340536 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:39.340549 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:39.340555 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:20:39.340712 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 19:20:39.341096 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":326,"threads_started":1}
I20260810 19:20:39.343607 23477 heartbeater.cc:347] Connected to a master server at 127.7.43.62:45021
I20260810 19:20:39.343896 23477 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:39.344645 23477 heartbeater.cc:511] Master 127.7.43.62:45021 requested a full tablet report, sending...
I20260810 19:20:39.346020 22929 ts_manager.cc:194] Registered new tserver with Master: 3cd6718a6c17468ca4e6f92ca4ab4c85 (127.7.43.2:39403)
I20260810 19:20:39.346792 22929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.2:44395
W20260810 19:20:39.670228 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:39.670493 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:39.670632 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:39.677901 23481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:39.678001 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:39.678138 23481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:39.687008 23481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:44193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:39.688243 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:39.690021 23481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:39.702953 23494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:39.702998 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:39.703243 23492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:39.703701 23481 server_base.cc:1034] running on GCE node
I20260810 19:20:39.704247 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:39.705178 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:39.706388 23481 hybrid_clock.cc:648] HybridClock initialized: now 1786389639706323 us; error 47 us; skew 500 ppm
I20260810 19:20:39.706691 23481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:39.708905 23481 webserver.cc:460] Webserver started at http://127.7.43.3:46657/ using document root <none> and password file <none>
I20260810 19:20:39.709635 23481 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:39.709748 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:39.710069 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:39.712882 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "0874d058e05b4371a5a870b18fee5666"
format_stamp: "Formatted at 2026-08-10 19:20:39 on dist-test-slave-7lbf"
server_key: "914f21817689de3916432bc37866c6e7"
server_key_iv: "fb792a7f4f924c1d3a65ddfda686f81c"
server_key_version: "encryptionkey@0"
I20260810 19:20:39.713704 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "0874d058e05b4371a5a870b18fee5666"
format_stamp: "Formatted at 2026-08-10 19:20:39 on dist-test-slave-7lbf"
server_key: "914f21817689de3916432bc37866c6e7"
server_key_iv: "fb792a7f4f924c1d3a65ddfda686f81c"
server_key_version: "encryptionkey@0"
I20260810 19:20:39.718622 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:20:39.721916 23505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:39.723202 23481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:20:39.723356 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "0874d058e05b4371a5a870b18fee5666"
format_stamp: "Formatted at 2026-08-10 19:20:39 on dist-test-slave-7lbf"
server_key: "914f21817689de3916432bc37866c6e7"
server_key_iv: "fb792a7f4f924c1d3a65ddfda686f81c"
server_key_version: "encryptionkey@0"
I20260810 19:20:39.723552 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:39.739945 23481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:39.740810 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:39.741073 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:39.742043 23481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:39.743525 23481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:39.743619 23481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:39.743729 23481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:39.743799 23481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:39.774906 23481 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:34533
I20260810 19:20:39.774967 23683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:34533 every 8 connection(s)
I20260810 19:20:39.776247 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:39.784195 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 23481
I20260810 19:20:39.784394 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:20:39.784816 7340 external_mini_cluster.cc:1428] Setting key bb650bab5ca3f4133c6901e9524ceccd
I20260810 19:20:39.788546 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:39.778733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:49895 (local address 127.7.43.62:45021)
0810 19:20:39.778974 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:39.778981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:39.780333 (+ 1352us) server_negotiation.cc:407] Connection header received
0810 19:20:39.782065 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:39.782069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:39.782138 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:39.782225 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:39.783212 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.784021 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.785878 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.786379 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.786492 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:39.787782 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:39.787788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:39.787792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:39.787809 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:39.787839 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:39.787851 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:39.787923 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:39.788134 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 19:20:39.788228 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 19:20:39.789572 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:39.779295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45021 (local address 127.7.43.3:49895)
0810 19:20:39.780184 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:39.780226 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:39.781820 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:39.782412 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:39.782420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:39.782505 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:39.783082 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:39.783099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.784169 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.784172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:39.785744 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:39.785760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.786599 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.786607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:39.786831 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:39.787483 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:20:39.787485 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:39.787523 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:20:39.787597 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:39.787935 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:39.787945 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:39.787954 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 19:20:39.788111 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 19:20:39.788500 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":275,"threads_started":1}
I20260810 19:20:39.790767 23685 heartbeater.cc:347] Connected to a master server at 127.7.43.62:45021
I20260810 19:20:39.791035 23685 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:39.791779 23685 heartbeater.cc:511] Master 127.7.43.62:45021 requested a full tablet report, sending...
I20260810 19:20:39.793020 22929 ts_manager.cc:194] Registered new tserver with Master: 0874d058e05b4371a5a870b18fee5666 (127.7.43.3:34533)
I20260810 19:20:39.793818 22929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.3:49895
I20260810 19:20:39.799821 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 19:20:39.805697 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:39.804274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.7.43.62:45021)
0810 19:20:39.804418 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:39.804422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:39.804869 (+ 447us) server_negotiation.cc:407] Connection header received
0810 19:20:39.805017 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:39.805021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:39.805087 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:39.805163 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:39.805476 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49564: BlockingRecv error: recv got EOF from 127.0.0.1:49564 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:20:39.806403 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 23025
I20260810 19:20:39.813349 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:39.807304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.7.43.1:34097)
0810 19:20:39.807815 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:39.807833 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:39.807858 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:20:39.807917 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:39.807923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:39.808109 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:39.808229 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:39.808517 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.809480 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.810664 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.811195 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.811287 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:39.812133 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:39.812142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:39.812154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:39.812726 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:39.812770 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:39.812795 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:39.812879 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:39.812955 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 19:20:39.813073 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":255,"threads_started":1}
I20260810 19:20:39.814529 23241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:39.899278 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 23025
I20260810 19:20:39.923202 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 23253
I20260810 19:20:39.930825 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:39.924069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.7.43.2:39403)
0810 19:20:39.924622 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:39.924628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:39.924657 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:20:39.924721 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:39.924727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:39.924960 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:39.925095 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:39.925406 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.926515 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.927842 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:39.928440 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:39.928535 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:39.929450 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:39.929459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:39.929472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:39.930111 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:39.930159 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:39.930178 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:39.930267 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:39.930392 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 19:20:39.930520 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
I20260810 19:20:39.932044 23463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:40.012516 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 23253
I20260810 19:20:40.038538 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 23481
I20260810 19:20:40.046330 23705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:40.039513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.7.43.3:34533)
0810 19:20:40.040140 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:40.040147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:40.040176 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:20:40.040239 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:40.040246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:40.040457 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:40.040587 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:40.040999 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:40.042059 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:40.043320 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:40.043870 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:40.043979 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:40.044855 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:40.044864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:40.044878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:40.045468 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:40.045691 (+ 223us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:40.045714 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:40.045798 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:40.045900 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 19:20:40.046031 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":306,"threads_started":1}
I20260810 19:20:40.047534 23678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:40.127395 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 23481
I20260810 19:20:40.152200 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 22878
I20260810 19:20:40.153415 22980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:40.255211 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 22878
2026-08-10T19:20:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2445 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T19:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:40Z Disabled control of system clock
I20260810 19:20:40.341835 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:34647
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 19:20:40.680049 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:40.680349 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:40.680444 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:40.687666 23713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:40.687769 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:40.687814 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:40.687879 23713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:40.687924 23713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:40.697108 23713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:34647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:40.698402 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:40.700170 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:40.711951 23720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:40.712016 23719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:40.711925 23722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:40.712803 23713 server_base.cc:1034] running on GCE node
I20260810 19:20:40.713630 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:40.715047 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:40.716266 23713 hybrid_clock.cc:648] HybridClock initialized: now 1786389640716216 us; error 43 us; skew 500 ppm
I20260810 19:20:40.716588 23713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:40.718845 23713 webserver.cc:460] Webserver started at http://127.7.43.62:38071/ using document root <none> and password file <none>
I20260810 19:20:40.719512 23713 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:40.719600 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:40.719864 23713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:40.722736 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "06a533bb961449dabcac4d3da5574a86"
format_stamp: "Formatted at 2026-08-10 19:20:40 on dist-test-slave-7lbf"
server_key: "18b775b68c272077869e5b69132e0a5c"
server_key_iv: "7b6cd6a8f6129f071cbfe3c606d64e7f"
server_key_version: "encryptionkey@0"
I20260810 19:20:40.723491 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "06a533bb961449dabcac4d3da5574a86"
format_stamp: "Formatted at 2026-08-10 19:20:40 on dist-test-slave-7lbf"
server_key: "18b775b68c272077869e5b69132e0a5c"
server_key_iv: "7b6cd6a8f6129f071cbfe3c606d64e7f"
server_key_version: "encryptionkey@0"
I20260810 19:20:40.728561 23713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:20:40.732008 23729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:40.733312 23713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:20:40.733455 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "06a533bb961449dabcac4d3da5574a86"
format_stamp: "Formatted at 2026-08-10 19:20:40 on dist-test-slave-7lbf"
server_key: "18b775b68c272077869e5b69132e0a5c"
server_key_iv: "7b6cd6a8f6129f071cbfe3c606d64e7f"
server_key_version: "encryptionkey@0"
I20260810 19:20:40.733698 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:40.739086 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:40.739778 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:40.740036 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:40.756724 23713 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:34647
I20260810 19:20:40.756754 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:34647 every 8 connection(s)
I20260810 19:20:40.758121 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:20:40.761849 23818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:40.768383 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 23713
I20260810 19:20:40.768582 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:20:40.769418 7340 external_mini_cluster.cc:1428] Setting key 329d5f9ca60d0a5dacb4714339042076
I20260810 19:20:40.772919 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86: Bootstrap starting.
I20260810 19:20:40.776361 23818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:40.777575 23818 log.cc:824] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:40.777714 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:40.771535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.7.43.62:34647)
0810 19:20:40.772486 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:40.772497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:40.772539 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:20:40.774273 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:40.774300 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:40.774739 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:40.775060 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:40.776126 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:40.776132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:40.776145 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:40.776158 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:40.776216 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:40.776232 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:40.776376 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:40.776563 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 19:20:40.776776 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":316,"threads_started":1}
I20260810 19:20:40.780779 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86: No bootstrap required, opened a new log
I20260810 19:20:40.789350 23818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a533bb961449dabcac4d3da5574a86" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 34647 } }
I20260810 19:20:40.789867 23818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:40.790005 23818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a533bb961449dabcac4d3da5574a86, State: Initialized, Role: FOLLOWER
I20260810 19:20:40.790629 23818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a533bb961449dabcac4d3da5574a86" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 34647 } }
I20260810 19:20:40.790856 23818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:20:40.790982 23818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:20:40.791138 23818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:40.793206 23818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a533bb961449dabcac4d3da5574a86" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 34647 } }
I20260810 19:20:40.793725 23818 leader_election.cc:302] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a533bb961449dabcac4d3da5574a86; no voters:
I20260810 19:20:40.794309 23818 leader_election.cc:288] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:20:40.794409 23823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:40.795073 23823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [term 1 LEADER]: Becoming Leader. State: Replica: 06a533bb961449dabcac4d3da5574a86, State: Running, Role: LEADER
I20260810 19:20:40.795692 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a533bb961449dabcac4d3da5574a86" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 34647 } }
I20260810 19:20:40.796360 23818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:40.798748 23831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a533bb961449dabcac4d3da5574a86. Latest consensus state: current_term: 1 leader_uuid: "06a533bb961449dabcac4d3da5574a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a533bb961449dabcac4d3da5574a86" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 34647 } } }
I20260810 19:20:40.798714 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a533bb961449dabcac4d3da5574a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a533bb961449dabcac4d3da5574a86" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 34647 } } }
I20260810 19:20:40.799243 23831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:40.799278 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:40.800148 23840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:40.806710 23840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:40.815548 23840 catalog_manager.cc:1347] Generated new cluster ID: a51e5d9ccb7c4ffba52e42d4ce3aeb4f
I20260810 19:20:40.815662 23840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:40.837973 23840 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:20:40.838421 23840 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:40.850033 23840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a533bb961449dabcac4d3da5574a86: Generated new TSK 0
I20260810 19:20:40.850839 23840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:20:40.858101 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:34647
--builtin_ntp_servers=127.7.43.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 19:20:41.197213 23852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:41.197497 23852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:41.197656 23852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:41.204905 23852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:41.205015 23852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:41.205169 23852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:41.214339 23852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:41.215678 23852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:41.217557 23852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:41.230741 23861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:41.230899 23866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:41.230767 23863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:41.232219 23852 server_base.cc:1034] running on GCE node
I20260810 19:20:41.232851 23852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:41.233861 23852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:41.235067 23852 hybrid_clock.cc:648] HybridClock initialized: now 1786389641235016 us; error 44 us; skew 500 ppm
I20260810 19:20:41.235375 23852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:41.237689 23852 webserver.cc:460] Webserver started at http://127.7.43.1:39295/ using document root <none> and password file <none>
I20260810 19:20:41.238423 23852 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:41.238536 23852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:41.238868 23852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:41.244011 23852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "5f5e510632c745498845b7340e1046c6"
format_stamp: "Formatted at 2026-08-10 19:20:41 on dist-test-slave-7lbf"
server_key: "ddf33335da5877f3156d8047e5061150"
server_key_iv: "cc912933c2dc767d830d55eeca0fd7cf"
server_key_version: "encryptionkey@0"
I20260810 19:20:41.244798 23852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "5f5e510632c745498845b7340e1046c6"
format_stamp: "Formatted at 2026-08-10 19:20:41 on dist-test-slave-7lbf"
server_key: "ddf33335da5877f3156d8047e5061150"
server_key_iv: "cc912933c2dc767d830d55eeca0fd7cf"
server_key_version: "encryptionkey@0"
I20260810 19:20:41.249943 23852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:20:41.253340 23877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:41.254628 23852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:20:41.254783 23852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "5f5e510632c745498845b7340e1046c6"
format_stamp: "Formatted at 2026-08-10 19:20:41 on dist-test-slave-7lbf"
server_key: "ddf33335da5877f3156d8047e5061150"
server_key_iv: "cc912933c2dc767d830d55eeca0fd7cf"
server_key_version: "encryptionkey@0"
I20260810 19:20:41.254985 23852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:41.260496 23852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:41.261262 23852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:41.261572 23852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:41.262465 23852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:41.263839 23852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:41.263937 23852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:41.264056 23852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:41.264127 23852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:41.294513 23852 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34215
I20260810 19:20:41.294565 24078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34215 every 8 connection(s)
I20260810 19:20:41.295827 23852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:41.297399 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 23852
I20260810 19:20:41.297614 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:20:41.298334 7340 external_mini_cluster.cc:1428] Setting key f7d9191ff0725dd93f47aa6dcf2c3b7a
I20260810 19:20:41.301301 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:34647
--builtin_ntp_servers=127.7.43.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 19:20:41.304401 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:41.298510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:50797 (local address 127.7.43.62:34647)
0810 19:20:41.299131 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:41.299138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:41.300358 (+ 1220us) server_negotiation.cc:407] Connection header received
0810 19:20:41.302158 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:41.302164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:41.302232 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:41.302332 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:41.303627 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:41.303634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:41.303639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:41.303658 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:41.303720 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:41.303733 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:41.303847 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:41.304044 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 19:20:41.304112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":377,"threads_started":1}
I20260810 19:20:41.305366 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:41.299134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:34647 (local address 127.7.43.1:50797)
0810 19:20:41.300202 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:41.300276 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:41.301884 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:41.302598 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:41.302606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:41.302703 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:41.303325 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:20:41.303327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:41.303366 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:20:41.303457 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:41.303829 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:41.303840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:41.303848 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:20:41.304126 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 19:20:41.304387 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":321,"threads_started":1}
I20260810 19:20:41.306870 24081 heartbeater.cc:347] Connected to a master server at 127.7.43.62:34647
I20260810 19:20:41.307189 24081 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:41.307902 24081 heartbeater.cc:511] Master 127.7.43.62:34647 requested a full tablet report, sending...
I20260810 19:20:41.310105 23754 ts_manager.cc:194] Registered new tserver with Master: 5f5e510632c745498845b7340e1046c6 (127.7.43.1:34215)
W20260810 19:20:41.636368 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:41.636662 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:41.636797 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:41.644119 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:41.644238 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:41.644390 24086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:41.653595 24086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:41.654896 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:41.656688 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:41.669633 24093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:41.669612 24096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:41.669664 24101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:41.670374 24086 server_base.cc:1034] running on GCE node
I20260810 19:20:41.671015 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:41.672127 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:41.673357 24086 hybrid_clock.cc:648] HybridClock initialized: now 1786389641673288 us; error 51 us; skew 500 ppm
I20260810 19:20:41.673740 24086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:41.676025 24086 webserver.cc:460] Webserver started at http://127.7.43.2:37115/ using document root <none> and password file <none>
I20260810 19:20:41.676744 24086 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:41.676862 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:41.677194 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:41.680124 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "8aa4a88026a54166aaf4f8a82948f9fa"
format_stamp: "Formatted at 2026-08-10 19:20:41 on dist-test-slave-7lbf"
server_key: "e59eb53618a1b2df06690958f1a226ec"
server_key_iv: "641c7d7d0c2d2ab70b8745c2b58e9f4a"
server_key_version: "encryptionkey@0"
I20260810 19:20:41.680948 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "8aa4a88026a54166aaf4f8a82948f9fa"
format_stamp: "Formatted at 2026-08-10 19:20:41 on dist-test-slave-7lbf"
server_key: "e59eb53618a1b2df06690958f1a226ec"
server_key_iv: "641c7d7d0c2d2ab70b8745c2b58e9f4a"
server_key_version: "encryptionkey@0"
I20260810 19:20:41.686102 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 19:20:41.689600 24108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:41.690902 24086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:41.691064 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "8aa4a88026a54166aaf4f8a82948f9fa"
format_stamp: "Formatted at 2026-08-10 19:20:41 on dist-test-slave-7lbf"
server_key: "e59eb53618a1b2df06690958f1a226ec"
server_key_iv: "641c7d7d0c2d2ab70b8745c2b58e9f4a"
server_key_version: "encryptionkey@0"
I20260810 19:20:41.691263 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:41.696977 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:41.697870 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:41.698113 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:41.699118 24086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:41.700700 24086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:41.700795 24086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:41.700906 24086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:41.700985 24086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:41.731419 24086 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:35535
I20260810 19:20:41.731483 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:35535 every 8 connection(s)
I20260810 19:20:41.732738 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:41.740512 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:41.735293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:55659 (local address 127.7.43.62:34647)
0810 19:20:41.735504 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:41.735510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:41.736895 (+ 1385us) server_negotiation.cc:407] Connection header received
0810 19:20:41.738607 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:41.738612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:41.738666 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:41.738751 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:41.739876 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:41.739880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:41.739883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:41.739898 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:41.739928 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:41.739939 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:41.740044 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:41.740159 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 19:20:41.740209 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:20:41.740686 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 24086
I20260810 19:20:41.740854 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:20:41.741322 7340 external_mini_cluster.cc:1428] Setting key cfb49f1c328b98f52c432372db880cc6
I20260810 19:20:41.741358 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:41.735830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:34647 (local address 127.7.43.2:55659)
0810 19:20:41.736751 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:41.736792 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:41.738368 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:41.738938 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:41.738946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:41.739034 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:41.739611 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:20:41.739613 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:41.739649 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:20:41.739732 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:41.740013 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:41.740018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:41.740023 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:20:41.740220 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 19:20:41.740407 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":275,"threads_started":1}
I20260810 19:20:41.742661 24306 heartbeater.cc:347] Connected to a master server at 127.7.43.62:34647
I20260810 19:20:41.742952 24306 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:41.743530 24306 heartbeater.cc:511] Master 127.7.43.62:34647 requested a full tablet report, sending...
I20260810 19:20:41.744118 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:34647
--builtin_ntp_servers=127.7.43.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 19:20:41.744871 23752 ts_manager.cc:194] Registered new tserver with Master: 8aa4a88026a54166aaf4f8a82948f9fa (127.7.43.2:35535)
W20260810 19:20:42.076956 24309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:42.077266 24309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:42.077399 24309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:42.084566 24309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:42.084677 24309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:42.084824 24309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:42.093804 24309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:42.095201 24309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:42.097055 24309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:42.109854 24321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:42.109890 24317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:42.110134 24314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:42.110749 24309 server_base.cc:1034] running on GCE node
I20260810 19:20:42.111335 24309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:42.112239 24309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:42.113427 24309 hybrid_clock.cc:648] HybridClock initialized: now 1786389642113367 us; error 53 us; skew 500 ppm
I20260810 19:20:42.113788 24309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:20:42.116043 24309 webserver.cc:460] Webserver started at http://127.7.43.3:43829/ using document root <none> and password file <none>
I20260810 19:20:42.116729 24309 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:42.116819 24309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:42.117125 24309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:42.120048 24309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "4f39e77506b143c3bb3cc0b0f1623791"
format_stamp: "Formatted at 2026-08-10 19:20:42 on dist-test-slave-7lbf"
server_key: "d950bc8e2bd5e5256d7b45929e2fc8ad"
server_key_iv: "8dc9d6d4039a803053d2343ad0c50c26"
server_key_version: "encryptionkey@0"
I20260810 19:20:42.120811 24309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "4f39e77506b143c3bb3cc0b0f1623791"
format_stamp: "Formatted at 2026-08-10 19:20:42 on dist-test-slave-7lbf"
server_key: "d950bc8e2bd5e5256d7b45929e2fc8ad"
server_key_iv: "8dc9d6d4039a803053d2343ad0c50c26"
server_key_version: "encryptionkey@0"
I20260810 19:20:42.126089 24309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:20:42.129433 24330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:42.130842 24309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:42.131008 24309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "4f39e77506b143c3bb3cc0b0f1623791"
format_stamp: "Formatted at 2026-08-10 19:20:42 on dist-test-slave-7lbf"
server_key: "d950bc8e2bd5e5256d7b45929e2fc8ad"
server_key_iv: "8dc9d6d4039a803053d2343ad0c50c26"
server_key_version: "encryptionkey@0"
I20260810 19:20:42.131208 24309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:42.136752 24309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:42.137579 24309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:42.137847 24309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:42.138731 24309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:42.140087 24309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:42.140180 24309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:42.140287 24309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:42.140357 24309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:42.171605 24309 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:44589
I20260810 19:20:42.171677 24508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:44589 every 8 connection(s)
I20260810 19:20:42.172906 24309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:42.181074 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:42.175605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:54565 (local address 127.7.43.62:34647)
0810 19:20:42.175838 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:42.175843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:42.177244 (+ 1401us) server_negotiation.cc:407] Connection header received
0810 19:20:42.179027 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:42.179031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:42.179085 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:42.179171 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:42.180458 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:42.180462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:42.180465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:42.180476 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:42.180501 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:42.180511 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:42.180596 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:42.180809 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 19:20:42.180858 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:20:42.182132 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:42.176098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:34647 (local address 127.7.43.3:54565)
0810 19:20:42.177084 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:42.177124 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:42.178769 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:42.179382 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:42.179390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:42.179483 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:42.180177 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:20:42.180180 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:42.180220 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:20:42.180318 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:42.180619 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:42.180628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:42.180635 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:20:42.180892 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 19:20:42.181109 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":281,"threads_started":1}
I20260810 19:20:42.182696 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 24309
I20260810 19:20:42.182835 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:20:42.183310 7340 external_mini_cluster.cc:1428] Setting key f37a96a401ffcf0f47516fb8b405e287
I20260810 19:20:42.183506 24509 heartbeater.cc:347] Connected to a master server at 127.7.43.62:34647
I20260810 19:20:42.183813 24509 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:42.184506 24509 heartbeater.cc:511] Master 127.7.43.62:34647 requested a full tablet report, sending...
I20260810 19:20:42.185945 23754 ts_manager.cc:194] Registered new tserver with Master: 4f39e77506b143c3bb3cc0b0f1623791 (127.7.43.3:44589)
I20260810 19:20:42.198604 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 19:20:42.204641 24084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:20:42.203160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.7.43.62:34647)
0810 19:20:42.203370 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:42.203378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:42.203781 (+ 403us) server_negotiation.cc:407] Connection header received
0810 19:20:42.203979 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:42.203983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:42.204036 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:42.204115 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:42.204384 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60582: BlockingRecv error: recv got EOF from 127.0.0.1:60582 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:20:42.205684 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 23852
I20260810 19:20:42.209000 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:42.206629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.7.43.1:34215)
0810 19:20:42.207225 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:42.207233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:42.207258 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:20:42.207316 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:42.207325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:42.207530 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:42.207654 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:42.208428 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:42.208433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:42.208446 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:42.208462 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:42.208511 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:42.208526 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:42.208627 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:42.208678 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 19:20:42.208758 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
I20260810 19:20:42.210333 24070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:42.290050 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 23852
I20260810 19:20:42.315766 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 24086
I20260810 19:20:42.319379 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:42.316671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.7.43.2:35535)
0810 19:20:42.317193 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:42.317200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:42.317226 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:20:42.317289 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:42.317295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:42.317507 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:42.317683 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:42.318660 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:42.318666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:42.318678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:42.318695 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:42.318736 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:42.318750 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:42.318871 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:42.319013 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 19:20:42.319120 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260810 19:20:42.320713 24299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:42.401621 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 24086
I20260810 19:20:42.425289 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 24309
I20260810 19:20:42.429196 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:42.426439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.7.43.3:44589)
0810 19:20:42.427142 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:42.427156 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:42.427188 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:20:42.427251 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:42.427258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:42.427466 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:42.427591 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:42.428501 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:42.428508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:42.428519 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:20:42.428536 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:42.428576 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:20:42.428590 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:42.428719 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:42.428782 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 19:20:42.428897 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":393,"threads_started":1}
I20260810 19:20:42.430467 24497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:42.510187 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 24309
I20260810 19:20:42.531921 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 23713
I20260810 19:20:42.533190 23809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:42.633937 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 23713
2026-08-10T19:20:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2379 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:20:42 dist-test-slave-7lbf krb5kdc[24541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:20:42 dist-test-slave-7lbf krb5kdc[24541](info): set up 2 sockets
Aug 10 19:20:42 dist-test-slave-7lbf krb5kdc[24541](info): commencing operation
krb5kdc: starting...
W20260810 19:20:44.029992 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.349s user 0.005s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:20:44 dist-test-slave-7lbf krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:20:44.483424 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43791
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:45.044158 24558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:45.044454 24558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:45.044587 24558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:45.051760 24558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:20:45.051895 24558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:45.051977 24558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:45.052059 24558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:20:45.052121 24558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:20:45.061976 24558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:45.063310 24558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:45.064661 24558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:45.076293 24564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:45.076341 24568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:45.076365 24566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:45.077157 24558 server_base.cc:1034] running on GCE node
I20260810 19:20:45.078001 24558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:45.079469 24558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:45.080753 24558 hybrid_clock.cc:648] HybridClock initialized: now 1786389645080691 us; error 44 us; skew 500 ppm
I20260810 19:20:45.081125 24558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:45 dist-test-slave-7lbf krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389645, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:45.087237 24558 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:20:45.088805 24558 webserver.cc:460] Webserver started at http://127.7.43.62:38483/ using document root <none> and password file <none>
I20260810 19:20:45.089498 24558 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:45.089669 24558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:45.089984 24558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:45.093015 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "da08e52abea042cfbb100ee3dfee9868"
format_stamp: "Formatted at 2026-08-10 19:20:45 on dist-test-slave-7lbf"
server_key: "3b450bf6ce73178f5fdee603391af41a"
server_key_iv: "c02f353ee2badfb82e29e4c8eca71238"
server_key_version: "encryptionkey@0"
I20260810 19:20:45.093830 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "da08e52abea042cfbb100ee3dfee9868"
format_stamp: "Formatted at 2026-08-10 19:20:45 on dist-test-slave-7lbf"
server_key: "3b450bf6ce73178f5fdee603391af41a"
server_key_iv: "c02f353ee2badfb82e29e4c8eca71238"
server_key_version: "encryptionkey@0"
I20260810 19:20:45.099078 24558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:20:45.102500 24579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:45.103825 24558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:20:45.104012 24558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "da08e52abea042cfbb100ee3dfee9868"
format_stamp: "Formatted at 2026-08-10 19:20:45 on dist-test-slave-7lbf"
server_key: "3b450bf6ce73178f5fdee603391af41a"
server_key_iv: "c02f353ee2badfb82e29e4c8eca71238"
server_key_version: "encryptionkey@0"
I20260810 19:20:45.104197 24558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:45.119889 24558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:45.124112 24558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:45.124393 24558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:45.142560 24558 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:43791
I20260810 19:20:45.142601 24662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:43791 every 8 connection(s)
I20260810 19:20:45.144006 24558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:20:45.147907 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:45.152586 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 24558
I20260810 19:20:45.152817 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:20:45.153751 7340 external_mini_cluster.cc:1428] Setting key 116f21dce4593da575f4cc291330de30
I20260810 19:20:45.160323 24663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868: Bootstrap starting.
I20260810 19:20:45.164043 24663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:45.165261 24663 log.cc:824] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:45.168576 24663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868: No bootstrap required, opened a new log
Aug 10 19:20:45 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:45.176664 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:45.156027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.7.43.62:43791)
0810 19:20:45.156809 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:45.156824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:45.156867 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:20:45.158716 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:45.158742 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:45.159152 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:45.159466 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:45.161363 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:45.162741 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:45.164454 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:45.165187 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:45.165322 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:45.169726 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:45.169733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:45.169748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:45.169787 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:45.172823 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:45.173446 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:45.173453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:45.173459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:45.173667 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:45.174025 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:45.174030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:45.174034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:45.174562 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:45.174881 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:45.175307 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 19:20:45.175700 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":355,"threads_started":1}
I20260810 19:20:45.176616 24663 raft_consensus.cc:348] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da08e52abea042cfbb100ee3dfee9868" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43791 } }
I20260810 19:20:45.177016 24663 raft_consensus.cc:374] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:45.177114 24663 raft_consensus.cc:729] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da08e52abea042cfbb100ee3dfee9868, State: Initialized, Role: FOLLOWER
I20260810 19:20:45.177712 24663 consensus_queue.cc:260] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da08e52abea042cfbb100ee3dfee9868" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43791 } }
I20260810 19:20:45.177937 24663 raft_consensus.cc:388] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:20:45.178081 24663 raft_consensus.cc:482] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:20:45.178249 24663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:45.180586 24663 raft_consensus.cc:504] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da08e52abea042cfbb100ee3dfee9868" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43791 } }
I20260810 19:20:45.181053 24663 leader_election.cc:302] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da08e52abea042cfbb100ee3dfee9868; no voters:
I20260810 19:20:45.181645 24663 leader_election.cc:288] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:20:45.181758 24668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:45.182755 24668 raft_consensus.cc:686] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [term 1 LEADER]: Becoming Leader. State: Replica: da08e52abea042cfbb100ee3dfee9868, State: Running, Role: LEADER
I20260810 19:20:45.183286 24668 consensus_queue.cc:237] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da08e52abea042cfbb100ee3dfee9868" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43791 } }
I20260810 19:20:45.183674 24663 sys_catalog.cc:564] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:20:45.185719 24672 sys_catalog.cc:455] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da08e52abea042cfbb100ee3dfee9868. Latest consensus state: current_term: 1 leader_uuid: "da08e52abea042cfbb100ee3dfee9868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da08e52abea042cfbb100ee3dfee9868" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43791 } } }
I20260810 19:20:45.186015 24672 sys_catalog.cc:458] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:45.186520 24671 sys_catalog.cc:455] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da08e52abea042cfbb100ee3dfee9868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da08e52abea042cfbb100ee3dfee9868" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43791 } } }
I20260810 19:20:45.186832 24671 sys_catalog.cc:458] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868 [sys.catalog]: This master's current role is: LEADER
I20260810 19:20:45.186851 24683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:20:45.193392 24683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:20:45.202469 24683 catalog_manager.cc:1347] Generated new cluster ID: 0af3df11010b44f4b444fc6b362e4d3e
I20260810 19:20:45.202585 24683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:20:45.212637 24683 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:20:45.214555 24683 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:20:45.224885 24683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868: Generated new TSK 0
I20260810 19:20:45.226020 24683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:20:45.318691 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43791
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:45.682101 24713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:45.682399 24713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:45.682538 24713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:45.689957 24713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:45.690083 24713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:45.690230 24713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:20:45.700091 24713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:45.701388 24713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:45.702777 24713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:45.716445 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:45.716539 24725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:45.716518 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:45.717335 24713 server_base.cc:1034] running on GCE node
I20260810 19:20:45.718003 24713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:45.718940 24713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:45.720141 24713 hybrid_clock.cc:648] HybridClock initialized: now 1786389645720091 us; error 42 us; skew 500 ppm
I20260810 19:20:45.720451 24713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:45 dist-test-slave-7lbf krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389645, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:45.723776 24713 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:20:45.725373 24713 webserver.cc:460] Webserver started at http://127.7.43.1:33351/ using document root <none> and password file <none>
I20260810 19:20:45.726078 24713 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:45.726192 24713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:45.726495 24713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:45.729473 24713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "1ec5e167adeb49a3b66c3a533577e143"
format_stamp: "Formatted at 2026-08-10 19:20:45 on dist-test-slave-7lbf"
server_key: "9c3121a8121ee863591d1da10cb3674e"
server_key_iv: "136875b11af3cb217819101b64603bc6"
server_key_version: "encryptionkey@0"
I20260810 19:20:45.730300 24713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "1ec5e167adeb49a3b66c3a533577e143"
format_stamp: "Formatted at 2026-08-10 19:20:45 on dist-test-slave-7lbf"
server_key: "9c3121a8121ee863591d1da10cb3674e"
server_key_iv: "136875b11af3cb217819101b64603bc6"
server_key_version: "encryptionkey@0"
I20260810 19:20:45.735471 24713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:20:45.738858 24737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:45.740135 24713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:45.740271 24713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "1ec5e167adeb49a3b66c3a533577e143"
format_stamp: "Formatted at 2026-08-10 19:20:45 on dist-test-slave-7lbf"
server_key: "9c3121a8121ee863591d1da10cb3674e"
server_key_iv: "136875b11af3cb217819101b64603bc6"
server_key_version: "encryptionkey@0"
I20260810 19:20:45.740454 24713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:45.755422 24713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:45.759639 24713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:45.759923 24713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:45.760907 24713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:45.762388 24713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:45.762470 24713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:45.762588 24713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:45.762643 24713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:45.793867 24713 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:43497
I20260810 19:20:45.793910 24912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:43497 every 8 connection(s)
I20260810 19:20:45.795146 24713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:20:45.802832 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 24713
I20260810 19:20:45.803030 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:20:45.803723 7340 external_mini_cluster.cc:1428] Setting key b61b0b823834c2497337378b26994d64
Aug 10 19:20:45 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389645, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:45.814790 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:45.797619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:53261 (local address 127.7.43.62:43791)
0810 19:20:45.798046 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:45.798050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:45.798905 (+ 855us) server_negotiation.cc:407] Connection header received
0810 19:20:45.800687 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:45.800693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:45.800788 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:45.800888 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:45.802478 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:45.803247 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:45.804932 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:45.805355 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:45.805510 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:45.809407 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:45.809412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:45.809416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:45.809451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:45.811784 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:45.812494 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:45.812499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:45.812501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:45.812621 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:45.813045 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:45.813049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:45.813051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:45.813312 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:45.813446 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:45.814213 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 19:20:45.814436 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":220,"threads_started":1}
I20260810 19:20:45.815804 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:45.798075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43791 (local address 127.7.43.1:53261)
0810 19:20:45.798750 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:45.798794 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:45.800414 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:45.801137 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:45.801146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:45.801721 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:45.802287 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:45.802305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:45.803399 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:45.803402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:45.804786 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:45.804799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:45.805622 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:45.805628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:45.805937 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:45.806628 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:45.806658 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:45.809178 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:45.811934 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:45.811943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:45.811959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:45.812367 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:45.812726 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:45.812731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:45.812734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:45.812929 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:45.813446 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:45.813454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:45.813822 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 19:20:45.814065 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 19:20:45.814618 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":275,"threads_started":1}
I20260810 19:20:45.817409 24913 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43791
I20260810 19:20:45.817739 24913 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:45.818500 24913 heartbeater.cc:511] Master 127.7.43.62:43791 requested a full tablet report, sending...
I20260810 19:20:45.820557 24608 ts_manager.cc:194] Registered new tserver with Master: 1ec5e167adeb49a3b66c3a533577e143 (127.7.43.1:43497)
I20260810 19:20:45.822032 24608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:53261
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:20:45.867118 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43791
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:46.204375 24922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:46.204665 24922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:46.204770 24922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:46.212287 24922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:46.212409 24922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:46.212529 24922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:20:46.222662 24922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:46.224009 24922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:46.225335 24922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:46.238947 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:46.238976 24935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:46.238893 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:46.239629 24922 server_base.cc:1034] running on GCE node
I20260810 19:20:46.240231 24922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:46.241142 24922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:46.242365 24922 hybrid_clock.cc:648] HybridClock initialized: now 1786389646242307 us; error 48 us; skew 500 ppm
I20260810 19:20:46.242691 24922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:46 dist-test-slave-7lbf krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:46.246048 24922 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:20:46.247547 24922 webserver.cc:460] Webserver started at http://127.7.43.2:35193/ using document root <none> and password file <none>
I20260810 19:20:46.248314 24922 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:46.248437 24922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:46.248739 24922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:46.251780 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4"
format_stamp: "Formatted at 2026-08-10 19:20:46 on dist-test-slave-7lbf"
server_key: "e0d1a2bc37c00c335c402f62fba572d1"
server_key_iv: "70e8bd660a446e7a08858901d0bb0a0d"
server_key_version: "encryptionkey@0"
I20260810 19:20:46.252552 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4"
format_stamp: "Formatted at 2026-08-10 19:20:46 on dist-test-slave-7lbf"
server_key: "e0d1a2bc37c00c335c402f62fba572d1"
server_key_iv: "70e8bd660a446e7a08858901d0bb0a0d"
server_key_version: "encryptionkey@0"
I20260810 19:20:46.257727 24922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 19:20:46.261209 24943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:46.262575 24922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:20:46.262730 24922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4"
format_stamp: "Formatted at 2026-08-10 19:20:46 on dist-test-slave-7lbf"
server_key: "e0d1a2bc37c00c335c402f62fba572d1"
server_key_iv: "70e8bd660a446e7a08858901d0bb0a0d"
server_key_version: "encryptionkey@0"
I20260810 19:20:46.262928 24922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:46.301774 24922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:46.305812 24922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:46.306098 24922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:46.307139 24922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:46.308550 24922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:46.308641 24922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:46.308769 24922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:46.308825 24922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:46.341154 24922 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:46109
I20260810 19:20:46.341207 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:46109 every 8 connection(s)
I20260810 19:20:46.342478 24922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:20:46.346810 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 24922
I20260810 19:20:46.347014 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:20:46.347589 7340 external_mini_cluster.cc:1428] Setting key cafb88961dea2619766a0548d18f58fb
Aug 10 19:20:46 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:46.364302 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.345091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:49655 (local address 127.7.43.62:43791)
0810 19:20:46.345558 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:46.345565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:46.346608 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 19:20:46.348849 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:46.348855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:46.348937 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:46.349047 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:46.351019 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.351720 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.353813 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.354242 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.354340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.358304 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:46.358309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:46.358312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:46.358347 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:46.360960 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:46.361882 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:46.361886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:46.361889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:46.362040 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:46.362502 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:46.362505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:46.362507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:46.362831 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:46.362994 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:46.363677 (+ 683us) server_negotiation.cc:299] Negotiation successful
0810 19:20:46.363944 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260810 19:20:46.365064 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.345698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43791 (local address 127.7.43.2:49655)
0810 19:20:46.346396 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:46.346464 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:46.348582 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:46.349284 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:46.349293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:46.349996 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:46.350826 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.350850 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.351864 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.351869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.353674 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.353688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.354459 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.354466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.354735 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.355423 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:46.355458 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:46.358047 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:46.361114 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:46.361123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:46.361141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:46.361748 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:46.362149 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:46.362153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:46.362156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:46.362397 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:46.362951 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:46.362959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:46.363295 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 19:20:46.363533 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 19:20:46.363940 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
I20260810 19:20:46.366570 25144 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43791
I20260810 19:20:46.366863 25144 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:46.367662 25144 heartbeater.cc:511] Master 127.7.43.62:43791 requested a full tablet report, sending...
I20260810 19:20:46.369135 24608 ts_manager.cc:194] Registered new tserver with Master: b8449f65465e4ae48fcb3cdc3ad1b1e4 (127.7.43.2:46109)
I20260810 19:20:46.369982 24608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:49655
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:20:46.417016 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43791
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:20:46.758669 25153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:20:46.759269 25153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:20:46.759573 25153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:20:46.767531 25153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:20:46.767679 25153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:20:46.767848 25153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:20:46.778025 25153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:20:46.779424 25153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:20:46.780757 25153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:20:46.795506 25163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:46.796015 25169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:20:46.796519 25165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:20:46.797086 25153 server_base.cc:1034] running on GCE node
I20260810 19:20:46.797710 25153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:20:46.798641 25153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:20:46.799844 25153 hybrid_clock.cc:648] HybridClock initialized: now 1786389646799785 us; error 52 us; skew 500 ppm
I20260810 19:20:46.800168 25153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:20:46 dist-test-slave-7lbf krb5kdc[24541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:20:46.803239 25153 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:20:46.804646 25153 webserver.cc:460] Webserver started at http://127.7.43.3:36229/ using document root <none> and password file <none>
I20260810 19:20:46.805347 25153 fs_manager.cc:359] Metadata directory not provided
I20260810 19:20:46.805465 25153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:20:46.805806 25153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:20:46.808815 25153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "816470ed4d7d4fb687eb9453604f217f"
format_stamp: "Formatted at 2026-08-10 19:20:46 on dist-test-slave-7lbf"
server_key: "7573304d11e4b10f83f51eda4ccf4044"
server_key_iv: "cc8a26ed15f8e65b4ebdf07df62a1621"
server_key_version: "encryptionkey@0"
I20260810 19:20:46.809623 25153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "816470ed4d7d4fb687eb9453604f217f"
format_stamp: "Formatted at 2026-08-10 19:20:46 on dist-test-slave-7lbf"
server_key: "7573304d11e4b10f83f51eda4ccf4044"
server_key_iv: "cc8a26ed15f8e65b4ebdf07df62a1621"
server_key_version: "encryptionkey@0"
I20260810 19:20:46.814729 25153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:20:46.818135 25178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:46.819450 25153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:20:46.819609 25153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "816470ed4d7d4fb687eb9453604f217f"
format_stamp: "Formatted at 2026-08-10 19:20:46 on dist-test-slave-7lbf"
server_key: "7573304d11e4b10f83f51eda4ccf4044"
server_key_iv: "cc8a26ed15f8e65b4ebdf07df62a1621"
server_key_version: "encryptionkey@0"
I20260810 19:20:46.819792 25153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:20:46.824936 24913 heartbeater.cc:503] Master 127.7.43.62:43791 was elected leader, sending a full tablet report...
I20260810 19:20:46.845458 25153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:20:46.849680 25153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:20:46.849936 25153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:20:46.850917 25153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:20:46.852314 25153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:20:46.852387 25153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:46.852490 25153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:20:46.852541 25153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:20:46.884610 25153 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:41937
I20260810 19:20:46.884708 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:41937 every 8 connection(s)
I20260810 19:20:46.885991 25153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:20:46.887468 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 25153
I20260810 19:20:46.887641 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:20:46.888180 7340 external_mini_cluster.cc:1428] Setting key 5f591a673bce9b25a9df34f066e56a6e
Aug 10 19:20:46 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:20:46.905961 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.888726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:60033 (local address 127.7.43.62:43791)
0810 19:20:46.889178 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:46.889184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:46.890406 (+ 1222us) server_negotiation.cc:407] Connection header received
0810 19:20:46.892557 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:46.892561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:46.892629 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:46.892714 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:46.894235 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.895033 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.896520 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.896916 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.897018 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.900315 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:46.900320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:46.900325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:46.900359 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:46.903100 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:46.903981 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:46.903986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:46.903988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:46.904116 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:46.904489 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:46.904492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:46.904494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:46.904751 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:46.904867 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:46.905471 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 19:20:46.905690 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260810 19:20:46.906913 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.889398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43791 (local address 127.7.43.3:60033)
0810 19:20:46.890238 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:46.890288 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:46.892309 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:46.892908 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:46.892916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:46.893406 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:20:46.894036 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.894054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.895140 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.895144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.896362 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.896374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.897107 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.897112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.897315 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.897969 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:20:46.897992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:20:46.900080 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:20:46.903256 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:46.903266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:46.903282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:46.903822 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:46.904204 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:20:46.904207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:20:46.904209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:20:46.904400 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:20:46.904844 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:20:46.904850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:20:46.905157 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 19:20:46.905376 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 19:20:46.905820 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":299,"threads_started":1}
I20260810 19:20:46.908254 25365 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43791
I20260810 19:20:46.908561 25365 heartbeater.cc:464] Registering TS with master...
I20260810 19:20:46.909343 25365 heartbeater.cc:511] Master 127.7.43.62:43791 requested a full tablet report, sending...
I20260810 19:20:46.910799 24608 ts_manager.cc:194] Registered new tserver with Master: 816470ed4d7d4fb687eb9453604f217f (127.7.43.3:41937)
I20260810 19:20:46.911586 24608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:60033
I20260810 19:20:46.915956 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:20:46.932421 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.920468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.7.43.62:43791)
0810 19:20:46.920671 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:46.920677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:46.920988 (+ 311us) server_negotiation.cc:407] Connection header received
0810 19:20:46.921121 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:46.921125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:46.921188 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:46.921263 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:46.922670 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.923404 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.924744 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.925189 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.925293 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.927526 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:46.927531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:46.927534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:46.927566 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:46.930064 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:46.930751 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:46.930754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:46.930757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:46.930865 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:46.931165 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:46.931169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:46.931170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:46.931495 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:46.931679 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:46.931951 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 19:20:46.932125 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:20:46.936623 24608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:20:46.943558 24608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:20:46.991564 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.956380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.7.43.3:41937)
0810 19:20:46.982554 (+ 26174us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:46.982564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:46.982603 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:20:46.982715 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:46.982724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:46.983026 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:46.983229 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:46.984750 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.986393 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.989063 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.990362 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.990505 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.990760 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:46.991040 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 19:20:46.991210 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25975,"thread_start_us":27077,"threads_started":1}
I20260810 19:20:46.991617 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.955681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:41937 (local address 127.0.0.1:40090)
0810 19:20:46.981813 (+ 26132us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:46.981832 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:46.981954 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:46.983564 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:46.983573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:46.983617 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:46.984437 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.984475 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.986565 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.986569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.988836 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.988863 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.990590 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.990596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.990790 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.990844 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 19:20:46.990977 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 19:20:46.991211 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25991,"thread_start_us":25848,"threads_started":1}
I20260810 19:20:46.992973 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.955994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:43497 (local address 127.0.0.1:33904)
0810 19:20:46.984705 (+ 28711us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:46.984722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:46.984838 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:46.985559 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:46.985564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:46.985589 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:46.985954 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.985967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.987887 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.987893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.990006 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.990020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.992099 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.992106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.992325 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.992356 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:20:46.992446 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:20:46.992576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28595,"thread_start_us":28168,"threads_started":1}
I20260810 19:20:46.993078 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.955443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:46109 (local address 127.0.0.1:35032)
0810 19:20:46.980799 (+ 25356us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:46.980869 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:46.981121 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:46.984812 (+ 3691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:46.984817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:46.984847 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:46.985239 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.985253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.987366 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.987371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.989734 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:46.989756 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.992268 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.992276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:46.992453 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.992485 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:20:46.992570 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 19:20:46.992743 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25024,"thread_start_us":28947,"threads_started":1}
I20260810 19:20:46.993716 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.955580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.7.43.2:46109)
0810 19:20:46.984221 (+ 28641us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:46.984229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:46.984260 (+ 31us) server_negotiation.cc:407] Connection header received
0810 19:20:46.984358 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:46.984365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:46.984602 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:46.984819 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:46.985919 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.987227 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.990323 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.991827 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.991959 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.992158 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:46.993191 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0810 19:20:46.993366 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28479,"thread_start_us":28255,"threads_started":1}
I20260810 19:20:46.993911 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:46.956543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.7.43.1:43497)
0810 19:20:46.979870 (+ 23327us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:46.979884 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:46.984990 (+ 5106us) server_negotiation.cc:407] Connection header received
0810 19:20:46.985107 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:46.985113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:46.985332 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:46.985568 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:46.986161 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.987729 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.990168 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:46.991538 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:46.991716 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:46.991943 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:46.992533 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 19:20:46.992748 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23072,"thread_start_us":23375,"threads_started":1}
I20260810 19:20:46.996127 25265 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3d0208ba6c4778bed74e4b901d170d (DEFAULT_TABLE table=test-table [id=7e06428515f343ed945384347be09e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:46.997370 25265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3d0208ba6c4778bed74e4b901d170d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:46.997340 25031 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3d0208ba6c4778bed74e4b901d170d (DEFAULT_TABLE table=test-table [id=7e06428515f343ed945384347be09e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:46.997920 24810 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3d0208ba6c4778bed74e4b901d170d (DEFAULT_TABLE table=test-table [id=7e06428515f343ed945384347be09e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:20:46.998795 25031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3d0208ba6c4778bed74e4b901d170d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:46.999293 24810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3d0208ba6c4778bed74e4b901d170d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:20:47.008935 25401 tablet_bootstrap.cc:492] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Bootstrap starting.
I20260810 19:20:47.012465 25401 tablet_bootstrap.cc:654] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:47.013866 25403 tablet_bootstrap.cc:492] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Bootstrap starting.
I20260810 19:20:47.013964 25401 log.cc:824] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:47.014672 25407 tablet_bootstrap.cc:492] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Bootstrap starting.
I20260810 19:20:47.017827 25403 tablet_bootstrap.cc:654] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:47.018464 25407 tablet_bootstrap.cc:654] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Neither blocks nor log segments found. Creating new log.
I20260810 19:20:47.019001 25403 log.cc:824] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:47.019764 25407 log.cc:824] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Log is configured to *not* fsync() on all Append() calls
I20260810 19:20:47.022042 25403 tablet_bootstrap.cc:492] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: No bootstrap required, opened a new log
I20260810 19:20:47.022310 25403 ts_tablet_manager.cc:1397] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260810 19:20:47.022891 25401 tablet_bootstrap.cc:492] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: No bootstrap required, opened a new log
I20260810 19:20:47.023149 25401 ts_tablet_manager.cc:1397] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260810 19:20:47.027484 25407 tablet_bootstrap.cc:492] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: No bootstrap required, opened a new log
I20260810 19:20:47.027766 25407 ts_tablet_manager.cc:1397] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260810 19:20:47.030261 25401 raft_consensus.cc:348] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.030704 25401 raft_consensus.cc:374] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:47.030807 25401 raft_consensus.cc:729] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816470ed4d7d4fb687eb9453604f217f, State: Initialized, Role: FOLLOWER
I20260810 19:20:47.031406 25401 consensus_queue.cc:260] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.031729 25403 raft_consensus.cc:348] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.032238 25403 raft_consensus.cc:374] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:47.032373 25403 raft_consensus.cc:729] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8449f65465e4ae48fcb3cdc3ad1b1e4, State: Initialized, Role: FOLLOWER
I20260810 19:20:47.033093 25403 consensus_queue.cc:260] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.033918 25365 heartbeater.cc:503] Master 127.7.43.62:43791 was elected leader, sending a full tablet report...
I20260810 19:20:47.034632 25401 ts_tablet_manager.cc:1428] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260810 19:20:47.035509 25144 heartbeater.cc:503] Master 127.7.43.62:43791 was elected leader, sending a full tablet report...
I20260810 19:20:47.036142 25403 ts_tablet_manager.cc:1428] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 19:20:47.038057 25407 raft_consensus.cc:348] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.038496 25407 raft_consensus.cc:374] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:20:47.038643 25407 raft_consensus.cc:729] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec5e167adeb49a3b66c3a533577e143, State: Initialized, Role: FOLLOWER
I20260810 19:20:47.039258 25407 consensus_queue.cc:260] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.041406 25407 ts_tablet_manager.cc:1428] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260810 19:20:47.047329 24916 tablet.cc:2369] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:20:47.094486 25145 tablet.cc:2369] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:20:47.137660 25367 tablet.cc:2369] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:20:47.280241 25415 raft_consensus.cc:482] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:20:47.280539 25415 raft_consensus.cc:504] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.282274 25415 leader_election.cc:288] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8449f65465e4ae48fcb3cdc3ad1b1e4 (127.7.43.2:46109), 1ec5e167adeb49a3b66c3a533577e143 (127.7.43.1:43497)
I20260810 19:20:47.289402 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.282863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:32777 (local address 127.7.43.1:43497)
0810 19:20:47.283097 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:47.283104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:47.283242 (+ 138us) server_negotiation.cc:407] Connection header received
0810 19:20:47.283467 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:47.283472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:47.283549 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:47.283623 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:47.284097 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.285082 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.287486 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.288654 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.288790 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.288897 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:47.289104 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 19:20:47.289219 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 19:20:47.289538 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.282749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:43497 (local address 127.7.43.3:32777)
0810 19:20:47.283135 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:47.283155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:47.283296 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:47.283684 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:47.283689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:47.283768 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:47.283936 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:47.283944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.285204 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.285208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:47.287320 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:47.287336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.288806 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.288810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:47.288949 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.288976 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:20:47.289064 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:20:47.289205 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":43,"spinlock_wait_cycles":26880}
I20260810 19:20:47.289842 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.282966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:35657 (local address 127.7.43.2:46109)
0810 19:20:47.283174 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:47.283181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:47.283745 (+ 564us) server_negotiation.cc:407] Connection header received
0810 19:20:47.283844 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:47.283849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:47.283914 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:47.283989 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:20:47.284509 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.285437 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.287719 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.288900 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.289045 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.289132 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:47.289457 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 19:20:47.289618 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:20:47.289911 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.282802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:46109 (local address 127.7.43.3:35657)
0810 19:20:47.283594 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 19:20:47.283610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:20:47.283728 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:20:47.284032 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:20:47.284036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:20:47.284061 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:20:47.284351 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:47.284359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.285601 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.285606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:47.287581 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:20:47.287593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.289131 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.289137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:20:47.289294 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.289327 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:20:47.289416 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 19:20:47.289559 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":20,"thread_start_us":450,"threads_started":1}
I20260810 19:20:47.290645 24845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3d0208ba6c4778bed74e4b901d170d" candidate_uuid: "816470ed4d7d4fb687eb9453604f217f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec5e167adeb49a3b66c3a533577e143" is_pre_election: true
I20260810 19:20:47.291030 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3d0208ba6c4778bed74e4b901d170d" candidate_uuid: "816470ed4d7d4fb687eb9453604f217f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" is_pre_election: true
I20260810 19:20:47.291157 24845 raft_consensus.cc:2454] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816470ed4d7d4fb687eb9453604f217f in term 0.
I20260810 19:20:47.291522 25067 raft_consensus.cc:2454] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816470ed4d7d4fb687eb9453604f217f in term 0.
I20260810 19:20:47.291937 25183 leader_election.cc:302] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec5e167adeb49a3b66c3a533577e143, 816470ed4d7d4fb687eb9453604f217f; no voters:
I20260810 19:20:47.292340 25415 raft_consensus.cc:2781] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:20:47.292500 25415 raft_consensus.cc:482] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:20:47.292603 25415 raft_consensus.cc:3037] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:47.294850 25415 raft_consensus.cc:504] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.295457 25415 leader_election.cc:288] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [CANDIDATE]: Term 1 election: Requested vote from peers b8449f65465e4ae48fcb3cdc3ad1b1e4 (127.7.43.2:46109), 1ec5e167adeb49a3b66c3a533577e143 (127.7.43.1:43497)
I20260810 19:20:47.295864 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3d0208ba6c4778bed74e4b901d170d" candidate_uuid: "816470ed4d7d4fb687eb9453604f217f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4"
I20260810 19:20:47.295941 24845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3d0208ba6c4778bed74e4b901d170d" candidate_uuid: "816470ed4d7d4fb687eb9453604f217f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec5e167adeb49a3b66c3a533577e143"
I20260810 19:20:47.296048 25067 raft_consensus.cc:3037] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:47.296144 24845 raft_consensus.cc:3037] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:20:47.298309 25067 raft_consensus.cc:2454] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816470ed4d7d4fb687eb9453604f217f in term 1.
I20260810 19:20:47.298455 24845 raft_consensus.cc:2454] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816470ed4d7d4fb687eb9453604f217f in term 1.
I20260810 19:20:47.298712 25181 leader_election.cc:302] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816470ed4d7d4fb687eb9453604f217f, b8449f65465e4ae48fcb3cdc3ad1b1e4; no voters:
I20260810 19:20:47.298996 25415 raft_consensus.cc:2781] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:20:47.299404 25415 raft_consensus.cc:686] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 1 LEADER]: Becoming Leader. State: Replica: 816470ed4d7d4fb687eb9453604f217f, State: Running, Role: LEADER
I20260810 19:20:47.299912 25415 consensus_queue.cc:237] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } }
I20260810 19:20:47.304386 24608 catalog_manager.cc:5495] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f reported cstate change: term changed from 0 to 1, leader changed from <none> to 816470ed4d7d4fb687eb9453604f217f (127.7.43.3). New cstate: current_term: 1 leader_uuid: "816470ed4d7d4fb687eb9453604f217f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816470ed4d7d4fb687eb9453604f217f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 } health_report { overall_health: UNKNOWN } } }
I20260810 19:20:47.359398 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.354607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.7.43.3:41937)
0810 19:20:47.354816 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:47.354821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:47.354839 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:20:47.355010 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:47.355014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:47.355088 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:47.355209 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:20:47.355507 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.356232 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.357484 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.357897 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.357988 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.358421 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:20:47.358934 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:20:47.359048 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:47.359084 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 19:20:47.359149 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:20:47.367383 25067 raft_consensus.cc:1261] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 1 FOLLOWER]: Refusing update from remote peer 816470ed4d7d4fb687eb9453604f217f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:47.367405 24845 raft_consensus.cc:1261] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 1 FOLLOWER]: Refusing update from remote peer 816470ed4d7d4fb687eb9453604f217f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:20:47.368381 25419 consensus_queue.cc:1035] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec5e167adeb49a3b66c3a533577e143" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 43497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:47.368755 25415 consensus_queue.cc:1035] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8449f65465e4ae48fcb3cdc3ad1b1e4" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:20:47.382138 25425 mvcc.cc:205] Tried to move back new op lower bound from 7317051995606716416 to 7317051995344101376. Current Snapshot: MvccSnapshot[applied={T|T < 7317051995606716416}]
I20260810 19:20:47.395855 24608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:47.396162 24608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:20:47.399943 24608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da08e52abea042cfbb100ee3dfee9868: Sending DeleteTablet for 3 replicas of tablet 1c3d0208ba6c4778bed74e4b901d170d
I20260810 19:20:47.401239 25265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3d0208ba6c4778bed74e4b901d170d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:47 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:40090
I20260810 19:20:47.401242 25031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3d0208ba6c4778bed74e4b901d170d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:47 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:35032
I20260810 19:20:47.401445 24810 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3d0208ba6c4778bed74e4b901d170d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:20:47 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:33904
I20260810 19:20:47.403015 25441 tablet_replica.cc:330] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: stopping tablet replica
I20260810 19:20:47.403053 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 24713
I20260810 19:20:47.403569 25441 raft_consensus.cc:2229] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:47.404186 25441 raft_consensus.cc:2258] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:47.406880 25441 ts_tablet_manager.cc:1905] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:47.414477 25441 ts_tablet_manager.cc:1918] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:47.414778 25441 log.cc:1196] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/1c3d0208ba6c4778bed74e4b901d170d
I20260810 19:20:47.415470 25441 ts_tablet_manager.cc:1939] T 1c3d0208ba6c4778bed74e4b901d170d P 1ec5e167adeb49a3b66c3a533577e143: Deleting consensus metadata
I20260810 19:20:47.417622 24583 catalog_manager.cc:4841] TS 1ec5e167adeb49a3b66c3a533577e143 (127.7.43.1:43497): tablet 1c3d0208ba6c4778bed74e4b901d170d (table test-table [id=7e06428515f343ed945384347be09e1d]) successfully deleted
I20260810 19:20:47.419765 25442 tablet_replica.cc:330] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: stopping tablet replica
I20260810 19:20:47.420250 25440 tablet_replica.cc:330] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: stopping tablet replica
I20260810 19:20:47.420279 25442 raft_consensus.cc:2229] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:20:47.420778 25440 raft_consensus.cc:2229] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:20:47.421133 25442 raft_consensus.cc:2258] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:20:47.421365 25440 raft_consensus.cc:2258] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 19:20:47 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:20:47.423517 25442 ts_tablet_manager.cc:1905] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:47.423565 25440 ts_tablet_manager.cc:1905] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:20:47.430734 25440 ts_tablet_manager.cc:1918] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:47.430986 25440 log.cc:1196] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/1c3d0208ba6c4778bed74e4b901d170d
I20260810 19:20:47.431500 25442 ts_tablet_manager.cc:1918] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:20:47.431612 25440 ts_tablet_manager.cc:1939] T 1c3d0208ba6c4778bed74e4b901d170d P b8449f65465e4ae48fcb3cdc3ad1b1e4: Deleting consensus metadata
I20260810 19:20:47.431723 25442 log.cc:1196] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/1c3d0208ba6c4778bed74e4b901d170d
I20260810 19:20:47.431751 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.406009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.7.43.1:43497)
0810 19:20:47.406198 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:47.406205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:47.410108 (+ 3903us) server_negotiation.cc:407] Connection header received
0810 19:20:47.411073 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:47.411078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:47.411166 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:47.411251 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:47.414609 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.415724 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.417565 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.418241 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.418338 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.423838 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:47.423854 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:47.423871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:47.423914 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:47.427564 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:47.428443 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:47.428454 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:47.428458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:47.428620 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:47.429099 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:47.429105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:47.429108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:47.429613 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:47.429894 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:47.431151 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0810 19:20:47.431375 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:20:47.432503 24904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:47.432525 25442 ts_tablet_manager.cc:1939] T 1c3d0208ba6c4778bed74e4b901d170d P 816470ed4d7d4fb687eb9453604f217f: Deleting consensus metadata
I20260810 19:20:47.433816 24581 catalog_manager.cc:4841] TS b8449f65465e4ae48fcb3cdc3ad1b1e4 (127.7.43.2:46109): tablet 1c3d0208ba6c4778bed74e4b901d170d (table test-table [id=7e06428515f343ed945384347be09e1d]) successfully deleted
I20260810 19:20:47.434279 24584 catalog_manager.cc:4841] TS 816470ed4d7d4fb687eb9453604f217f (127.7.43.3:41937): tablet 1c3d0208ba6c4778bed74e4b901d170d (table test-table [id=7e06428515f343ed945384347be09e1d]) successfully deleted
I20260810 19:20:47.521204 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 24713
I20260810 19:20:47.544262 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 24922
Aug 10 19:20:47 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:20:47.560657 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.545118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.7.43.2:46109)
0810 19:20:47.545279 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:47.545283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:47.545301 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:20:47.545482 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:47.545486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:47.545643 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:47.545727 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:47.547338 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.548056 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.549953 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.550333 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.550423 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.555321 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:47.555336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:47.555354 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:47.555388 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:47.558070 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:47.558806 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:47.558812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:47.558815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:47.558962 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:47.559321 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:47.559324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:47.559326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:47.559698 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:47.559911 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:47.560227 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 19:20:47.560385 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:20:47.561337 25138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:47.651206 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 24922
I20260810 19:20:47.675395 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 25153
Aug 10 19:20:47 dist-test-slave-7lbf krb5kdc[24541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:20:47.692119 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:20:47.676253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.7.43.3:41937)
0810 19:20:47.676455 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:20:47.676461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:20:47.676627 (+ 166us) server_negotiation.cc:407] Connection header received
0810 19:20:47.676811 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:20:47.676816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:20:47.676895 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:20:47.676981 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:20:47.678336 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.679146 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.680462 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:20:47.680969 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:20:47.681086 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:20:47.685626 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:20:47.685641 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:20:47.685658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:20:47.685698 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:20:47.688721 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:47.689825 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:47.689833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:47.689835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:47.689995 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:20:47.690449 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:20:47.690452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:20:47.690454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:20:47.690814 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:20:47.691077 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 19:20:47.691527 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 19:20:47.691732 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:20:47.693040 25357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:47.812731 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 25153
I20260810 19:20:47.832300 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 24558
I20260810 19:20:47.833459 24656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:20:47.941802 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 24558
2026-08-10T19:20:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5311 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:20:48 dist-test-slave-7lbf krb5kdc[25454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:20:48 dist-test-slave-7lbf krb5kdc[25454](info): set up 2 sockets
Aug 10 19:20:48 dist-test-slave-7lbf krb5kdc[25454](info): commencing operation
krb5kdc: starting...
W20260810 19:20:49.205039 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:20:49 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:20:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/rangeradmin_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/rangeradmin_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/rangerlookup_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/rangerlookup_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
I20260810 19:20:49.606472 7340 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuOTVHW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/postgres -l logfile start
2026-08-10 19:20:51.914 UTC [25509] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 19:20:51.914 UTC [25509] LOG: listening on IPv4 address "127.7.43.20", port 37085
2026-08-10 19:20:51.918 UTC [25509] LOG: listening on Unix socket "/tmp/.s.PGSQL.37085"
2026-08-10 19:20:51.937 UTC [25515] LOG: database system was shut down at 2026-08-10 19:20:50 UTC
2026-08-10 19:20:51.941 UTC [25509] LOG: database system is ready to accept connections
I20260810 19:20:53.049841 7340 mini_postgres.cc:96] Postgres bound to 37085
2026-08-10 19:20:53.074 UTC [25532] FATAL: database "slave" does not exist
127.7.43.20:37085 - accepting connections
I20260810 19:20:53.075407 7340 mini_ranger.cc:161] Starting Ranger...
I20260810 19:20:53.107888 7340 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 19:20:53.561842 7340 mini_ranger.cc:87] Created ranger Postgres database
I20260810 19:20:53.561996 7340 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-admin
2026-08-10 19:20:53,857 [I] DB FLAVOR :POSTGRES
2026-08-10 19:20:53,857 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:20:53,857 [I] Checking connection..
2026-08-10 19:20:53,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 19:20:54,127 [I] Checking connection passed.
2026-08-10 19:20:54,128 [I] --------- Verifying version history table ---------
2026-08-10 19:20:54,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:20:54,392 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:20:54,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:20:54,636 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:20:54,637 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 19:20:54,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 19:20:54.882 UTC [25639] WARNING: there is no transaction in progress
2026-08-10 19:20:54,893 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 19:20:54,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:20:55,159 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 19:20:55,159 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:20:55,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 19:20:55,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 19:20:55,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7lbf.c.gcp-upstream.internal','N') ;"
2026-08-10 19:20:55,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:20:56,199 [I] Table x_portal_user does not exist in database ranger
2026-08-10 19:20:56,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:20:56,455 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 19:20:56,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:20:56,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:20:56,969 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 19:20:56,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 19:20:57.922 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:57.938 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:57.954 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:57.969 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:57.985 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.002 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.057 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.063 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.078 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.088 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.098 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.112 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.122 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.129 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.136 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58.142 UTC [25848] WARNING: there is no transaction in progress
2026-08-10 19:20:58,749 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 19:20:58,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 19:20:59,011 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 19:20:59,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:20:59,270 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 19:20:59,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:20:59,533 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 19:20:59,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:20:59,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:21:00,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7lbf.c.gcp-upstream.internal';"
2026-08-10 19:21:00,279 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 19:21:00,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 19:21:00,525 [I] DB_PATCHES have already been applied
I20260810 19:21:00.531333 7340 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuOTVHW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 19:21:00.531457 7340 mini_ranger.cc:193] Using host: 127.7.43.20
I20260810 19:21:00.553860 7340 mini_ranger.cc:227] Ranger admin URL: http://127.7.43.20:40185
Aug 10, 2026 7:21:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:21:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:21:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:21:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:21:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:21:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 19:21:01.852211 7340 mini_ranger.cc:160] Time spent starting Ranger: real 8.777s user 0.000s sys 0.067s
Aug 10, 2026 7:21:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:21:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 7:21:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:21:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:21:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 19:21:23.825253 7340 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/rangerkms_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/rangerkms_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.43.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.43.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 19:21:24 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389684, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 19:21:24.051198 7340 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 19:21:24.081386 7340 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 19:21:24.929147 7340 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 19:21:24.929288 7340 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms
I20260810 19:21:24.929454 7340 mini_ranger_kms.cc:227] Using postgres at 127.7.43.20:37085
2026-08-10 19:21:25,492 [I] DB FLAVOR :POSTGRES
2026-08-10 19:21:25,493 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:21:25,493 [I] Checking connection
2026-08-10 19:21:25,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 19:21:25,738 [I] connection success
2026-08-10 19:21:25,738 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 19:21:25,738 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 19:21:25,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 19:21:26,013 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 19:21:26,013 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:21:26,013 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 19:21:26,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:37085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 19:21:26,281 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 19:21:26.800063 7340 mini_ranger_kms.cc:310] Created kms service
I20260810 19:21:26.980881 7340 mini_ranger_kms.cc:326] Created kudu user
I20260810 19:21:27.028237 7340 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 19:21:27 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389684, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:21:27.423482 7340 mini_ranger_kms.cc:384] Added ranger policy
I20260810 19:21:27.425138 7340 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 19:21:27.425223 7340 mini_ranger_kms.cc:240] Using host: 127.7.43.20
I20260810 19:21:27.453845 7340 mini_ranger_kms.cc:276] Ranger KMS PID: 26158
I20260810 19:21:27.454072 7340 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.43.20:46071
Aug 10, 2026 7:21:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:21:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:21:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:21:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:21:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:21:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 19:21:29 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786389689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.43.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:21:30.135830 7340 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.085s user 0.146s sys 0.425s
I20260810 19:21:30.136240 7340 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 19:21:30.136528 7340 mini_ranger_kms.cc:401] 127.7.43.20:46071/kms/v1/keys
Aug 10 19:21:30 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.43.20@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
Aug 10 19:21:30 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.43.20@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
Aug 10 19:21:33 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:21:33.257647 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42657
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:42657
--ranger_config_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:46071/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:21:33.906526 26250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:21:33.906932 26250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:21:33.907088 26250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:21:33.918257 26250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:21:33.918493 26250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:21:33.918627 26250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:21:33.918807 26250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:21:33.918962 26250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:21:33.936239 26250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.43.20:46071/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:42657
--ranger_config_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:21:33.937965 26250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:21:33.939654 26250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:21:33.955317 26260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:33.955536 26267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:33.955336 26262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:21:33.956497 26250 server_base.cc:1034] running on GCE node
I20260810 19:21:33.957469 26250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:21:33.959178 26250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:21:33.960409 26250 hybrid_clock.cc:648] HybridClock initialized: now 1786389693960355 us; error 57 us; skew 500 ppm
I20260810 19:21:33.960777 26250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:21:33 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:21:33.968220 26250 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:21:33.969914 26250 webserver.cc:460] Webserver started at http://127.7.43.62:43455/ using document root <none> and password file <none>
I20260810 19:21:33.970651 26250 fs_manager.cc:359] Metadata directory not provided
I20260810 19:21:33.970777 26250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:21:33.971086 26250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:21:33 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:21:34.028546 26250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "6d799ebc0f7b46229e7047543213971f"
format_stamp: "Formatted at 2026-08-10 19:21:34 on dist-test-slave-7lbf"
server_key: "93aaf44d892fb1d75953310507b05933"
server_key_iv: "57441d71777e5a2d54f41e4166a3b10f"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:34.029466 26250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "6d799ebc0f7b46229e7047543213971f"
format_stamp: "Formatted at 2026-08-10 19:21:34 on dist-test-slave-7lbf"
server_key: "93aaf44d892fb1d75953310507b05933"
server_key_iv: "57441d71777e5a2d54f41e4166a3b10f"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:34.035533 26250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 19:21:34.057318 26277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:34.059018 26250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:21:34.059201 26250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "6d799ebc0f7b46229e7047543213971f"
format_stamp: "Formatted at 2026-08-10 19:21:34 on dist-test-slave-7lbf"
server_key: "93aaf44d892fb1d75953310507b05933"
server_key_iv: "57441d71777e5a2d54f41e4166a3b10f"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:34.059420 26250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:21:34.078797 26250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:21:34.083113 26250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:21:34.083417 26250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:21:34.101701 26250 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:42657
I20260810 19:21:34.101720 26369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:42657 every 8 connection(s)
I20260810 19:21:34.103076 26250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:21:34.106928 26371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:21:34.109954 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 26250
I20260810 19:21:34.110150 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:21:34.110800 7340 external_mini_cluster.cc:1428] Setting key b980de67a3059bfd73791b2f2d9a7319
I20260810 19:21:34.119311 26371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f: Bootstrap starting.
I20260810 19:21:34.123272 26371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f: Neither blocks nor log segments found. Creating new log.
I20260810 19:21:34.124696 26371 log.cc:824] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:21:34 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:21:34.128000 26371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f: No bootstrap required, opened a new log
I20260810 19:21:34.134913 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:34.112707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.7.43.62:42657)
0810 19:21:34.113469 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:34.113482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:34.113551 (+ 69us) server_negotiation.cc:407] Connection header received
0810 19:21:34.115213 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:34.115236 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:34.115649 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:34.116014 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:34.117766 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:34.119122 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:34.120832 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:34.121590 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:34.121721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:34.126430 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:34.126439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:34.126460 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:34.126499 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:34.130079 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:34.130866 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:34.130876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:34.130886 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:34.131111 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:34.131457 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:34.131463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:34.131466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:34.132228 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:34.132564 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:34.132931 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 19:21:34.133405 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":350,"threads_started":1}
I20260810 19:21:34.138320 26371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d799ebc0f7b46229e7047543213971f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42657 } }
I20260810 19:21:34.138672 26371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:21:34.138759 26371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d799ebc0f7b46229e7047543213971f, State: Initialized, Role: FOLLOWER
I20260810 19:21:34.139261 26371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d799ebc0f7b46229e7047543213971f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42657 } }
I20260810 19:21:34.139482 26371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:21:34.139601 26371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:21:34.139757 26371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:21:34.142211 26371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d799ebc0f7b46229e7047543213971f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42657 } }
I20260810 19:21:34.142735 26371 leader_election.cc:302] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d799ebc0f7b46229e7047543213971f; no voters:
I20260810 19:21:34.143308 26371 leader_election.cc:288] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:21:34.143440 26377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:21:34.144095 26377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [term 1 LEADER]: Becoming Leader. State: Replica: 6d799ebc0f7b46229e7047543213971f, State: Running, Role: LEADER
I20260810 19:21:34.144616 26377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d799ebc0f7b46229e7047543213971f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42657 } }
I20260810 19:21:34.145404 26371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:21:34.147361 26378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d799ebc0f7b46229e7047543213971f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d799ebc0f7b46229e7047543213971f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42657 } } }
I20260810 19:21:34.147922 26378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [sys.catalog]: This master's current role is: LEADER
I20260810 19:21:34.149005 26379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d799ebc0f7b46229e7047543213971f. Latest consensus state: current_term: 1 leader_uuid: "6d799ebc0f7b46229e7047543213971f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d799ebc0f7b46229e7047543213971f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42657 } } }
I20260810 19:21:34.149652 26379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f [sys.catalog]: This master's current role is: LEADER
I20260810 19:21:34.152159 26387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:21:34.157827 26387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:21:34.158705 26371 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:21:34.180243 26387 catalog_manager.cc:1347] Generated new cluster ID: 70c47ac800cf440cbe75a40e817c3c84
I20260810 19:21:34.180410 26387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:21:34.205186 26387 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:21:34.207628 26387 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:21:34.218888 26387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f: Generated new TSK 0
I20260810 19:21:34.219830 26387 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:21:35.006842 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:46071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42657
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:21:35 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786389695, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 19:21:35.508812 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:21:35.509557 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:21:35.509753 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:21:35.526207 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:21:35.526530 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:21:35.526865 26453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:21:35.548403 26453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.43.20:46071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:21:35.550522 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:21:35.552657 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:21:35.611605 26468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:35.611559 26467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:35.613114 26472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:21:35.620965 26453 server_base.cc:1034] running on GCE node
I20260810 19:21:35.622251 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:21:35.624290 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:21:35.627005 26453 hybrid_clock.cc:648] HybridClock initialized: now 1786389695626890 us; error 105 us; skew 500 ppm
I20260810 19:21:35.627676 26453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:21:35 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389695, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:21:35.636776 26453 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:21:35.642186 26453 webserver.cc:460] Webserver started at http://127.7.43.1:40655/ using document root <none> and password file <none>
I20260810 19:21:35.643221 26453 fs_manager.cc:359] Metadata directory not provided
I20260810 19:21:35.643448 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:21:35.643963 26453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:21:35 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389695, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:21:35.669587 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "ad147ae992664ac4a1120512c60c6740"
format_stamp: "Formatted at 2026-08-10 19:21:35 on dist-test-slave-7lbf"
server_key: "d1c9fb47e8c5173c28965f03676b5397"
server_key_iv: "b58f29f8b86e95aad7a163350f5f4005"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:35.671393 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "ad147ae992664ac4a1120512c60c6740"
format_stamp: "Formatted at 2026-08-10 19:21:35 on dist-test-slave-7lbf"
server_key: "d1c9fb47e8c5173c28965f03676b5397"
server_key_iv: "b58f29f8b86e95aad7a163350f5f4005"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:35.682057 26453 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20260810 19:21:35.704631 26484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:35.706465 26453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:21:35.706722 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "ad147ae992664ac4a1120512c60c6740"
format_stamp: "Formatted at 2026-08-10 19:21:35 on dist-test-slave-7lbf"
server_key: "d1c9fb47e8c5173c28965f03676b5397"
server_key_iv: "b58f29f8b86e95aad7a163350f5f4005"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:35.707132 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:21:35.735431 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:21:35.741406 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:21:35.742105 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:21:35.746328 26453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:21:35.749660 26453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:21:35.749933 26453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 19:21:35.750273 26453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:21:35.750483 26453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:21:35 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389695, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:21:36.066182 26453 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34127
I20260810 19:21:36.066334 26681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34127 every 8 connection(s)
I20260810 19:21:36.068928 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:21:36.077775 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 26453
I20260810 19:21:36.077986 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:21:36.079017 7340 external_mini_cluster.cc:1428] Setting key fbe3d16dc2ef3d1602bc75294d4179bd
Aug 10 19:21:36 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389695, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:21:36.112982 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:36.075326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:39369 (local address 127.7.43.62:42657)
0810 19:21:36.088384 (+ 13058us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:36.088390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:36.088406 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:21:36.088465 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:36.088468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:36.088559 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:36.088662 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:36.091122 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:36.092257 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:36.095325 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:36.096159 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:36.096286 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:36.101379 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:36.101390 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:36.101395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:36.101438 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:36.106536 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:36.107604 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:36.107610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:36.107614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:36.107850 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:36.108516 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:36.108521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:36.108525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:36.108967 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:36.109109 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:36.112068 (+ 2959us) server_negotiation.cc:299] Negotiation successful
0810 19:21:36.112363 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12937,"thread_start_us":13482,"threads_started":1}
I20260810 19:21:36.113494 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:36.074322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42657 (local address 127.7.43.1:39369)
0810 19:21:36.084355 (+ 10033us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:36.084400 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:36.087552 (+ 3152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:36.089058 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:36.089068 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:36.089959 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:21:36.090918 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:36.090943 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:36.092531 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:36.092537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:36.095126 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:36.095152 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:36.096548 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:36.096559 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:36.096954 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:36.097841 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:21:36.097867 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:21:36.100978 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:21:36.106739 (+ 5761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:21:36.106751 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:21:36.106774 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:21:36.107437 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:21:36.108006 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:21:36.108012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:21:36.108015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:21:36.108375 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:21:36.109181 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:21:36.109190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:21:36.109726 (+ 536us) client_negotiation.cc:769] Sending connection context
0810 19:21:36.110100 (+ 374us) client_negotiation.cc:240] Negotiation successful
0810 19:21:36.110820 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9660,"thread_start_us":11099,"threads_started":1}
I20260810 19:21:36.115479 26682 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42657
I20260810 19:21:36.115876 26682 heartbeater.cc:464] Registering TS with master...
I20260810 19:21:36.116973 26682 heartbeater.cc:511] Master 127.7.43.62:42657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
I20260810 19:21:36.119622 26305 ts_manager.cc:194] Registered new tserver with Master: ad147ae992664ac4a1120512c60c6740 (127.7.43.1:34127)
I20260810 19:21:36.122112 26305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:39369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:21:36.157416 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:46071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42657
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:21:36 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389695, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
W20260810 19:21:36.929474 26696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:21:36.929920 26696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:21:36.930176 26696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:21:36.941258 26696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:21:36.941493 26696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:21:36.941763 26696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:21:36.956269 26696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.43.20:46071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:21:36.957875 26696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:21:36.959452 26696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:21:36.981312 26723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:36.982041 26720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:36.982211 26721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:21:36.983006 26696 server_base.cc:1034] running on GCE node
I20260810 19:21:36.983907 26696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:21:36.985210 26696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:21:36.986511 26696 hybrid_clock.cc:648] HybridClock initialized: now 1786389696986377 us; error 59 us; skew 500 ppm
I20260810 19:21:36.986977 26696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:21:36 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:21:36.991892 26696 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:21:36.994308 26696 webserver.cc:460] Webserver started at http://127.7.43.2:34057/ using document root <none> and password file <none>
I20260810 19:21:36.995247 26696 fs_manager.cc:359] Metadata directory not provided
I20260810 19:21:36.995472 26696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:21:36.995937 26696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:21:36 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:21:37.016544 26696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce"
format_stamp: "Formatted at 2026-08-10 19:21:37 on dist-test-slave-7lbf"
server_key: "c849932cdffa2ef887d3c54030215f1a"
server_key_iv: "2e147c6d9bdcad79ddad59bef8e2cf7a"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:37.017915 26696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce"
format_stamp: "Formatted at 2026-08-10 19:21:37 on dist-test-slave-7lbf"
server_key: "c849932cdffa2ef887d3c54030215f1a"
server_key_iv: "2e147c6d9bdcad79ddad59bef8e2cf7a"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:37.025506 26696 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260810 19:21:37.047531 26735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:37.049829 26696 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20260810 19:21:37.050153 26696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce"
format_stamp: "Formatted at 2026-08-10 19:21:37 on dist-test-slave-7lbf"
server_key: "c849932cdffa2ef887d3c54030215f1a"
server_key_iv: "2e147c6d9bdcad79ddad59bef8e2cf7a"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:37.050575 26696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:21:37.089715 26696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:21:37.096256 26696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:21:37.096832 26696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:21:37.098378 26696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:21:37.100982 26696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:21:37.101125 26696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:37.101343 26696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:21:37.101553 26696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 19:21:37.126912 26682 heartbeater.cc:503] Master 127.7.43.62:42657 was elected leader, sending a full tablet report...
I20260810 19:21:37.203796 26696 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:41633
I20260810 19:21:37.203975 26917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:41633 every 8 connection(s)
I20260810 19:21:37.205716 26696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:21:37.214545 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 26696
I20260810 19:21:37.214723 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:21:37.215281 7340 external_mini_cluster.cc:1428] Setting key e263b906f5d004d2adf9ef6a1a0b7530
Aug 10 19:21:37 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389696, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:21:37.238274 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:37.209776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:38617 (local address 127.7.43.62:42657)
0810 19:21:37.210683 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:37.210690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:37.212357 (+ 1667us) server_negotiation.cc:407] Connection header received
0810 19:21:37.216355 (+ 3998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:37.216363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:37.216458 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:37.216576 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:37.219121 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:37.220314 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:37.222743 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:37.223371 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:37.223506 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:37.228488 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:37.228494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:37.228498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:37.228538 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:37.233287 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:37.234580 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:37.234587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:37.234590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:37.234878 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:37.235567 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:37.235573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:37.235576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:37.236045 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:37.236199 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:37.237303 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0810 19:21:37.237642 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":538,"threads_started":1}
I20260810 19:21:37.239532 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:37.210441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42657 (local address 127.7.43.2:38617)
0810 19:21:37.212159 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:37.212216 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:37.215731 (+ 3515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:37.216940 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:37.216954 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:37.217933 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:21:37.218916 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:37.218942 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:37.220509 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:37.220513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:37.222561 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:37.222578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:37.223735 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:37.223743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:37.224052 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:37.224947 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:21:37.224976 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:21:37.228109 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:21:37.233486 (+ 5377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:21:37.233498 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:21:37.233545 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:21:37.234367 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:21:37.235018 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:21:37.235023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:21:37.235026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:21:37.235420 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:21:37.236196 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:21:37.236206 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:21:37.236723 (+ 517us) client_negotiation.cc:769] Sending connection context
0810 19:21:37.237074 (+ 351us) client_negotiation.cc:240] Negotiation successful
0810 19:21:37.237843 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":759,"threads_started":1}
I20260810 19:21:37.241796 26918 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42657
I20260810 19:21:37.242272 26918 heartbeater.cc:464] Registering TS with master...
I20260810 19:21:37.243719 26918 heartbeater.cc:511] Master 127.7.43.62:42657 requested a full tablet report, sending...
I20260810 19:21:37.246162 26305 ts_manager.cc:194] Registered new tserver with Master: a44ca59ce3014de0bc2e7b80fa1d61ce (127.7.43.2:41633)
I20260810 19:21:37.247455 26305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:38617
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:21:37.289742 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:46071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42657
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:21:37.963719 26927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:21:37.964178 26927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:21:37.964457 26927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:21:37.975454 26927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:21:37.975703 26927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:21:37.975957 26927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:21:37.990521 26927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.43.20:46071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:21:37.992436 26927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:21:37.994065 26927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:21:38.013020 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:38.013677 26943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:21:38.013877 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:21:38.015177 26927 server_base.cc:1034] running on GCE node
I20260810 19:21:38.015867 26927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:21:38.017104 26927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:21:38.018364 26927 hybrid_clock.cc:648] HybridClock initialized: now 1786389698018262 us; error 42 us; skew 500 ppm
I20260810 19:21:38.018770 26927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:21:38 dist-test-slave-7lbf krb5kdc[25454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389698, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:21:38.023310 26927 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:21:38.025506 26927 webserver.cc:460] Webserver started at http://127.7.43.3:38849/ using document root <none> and password file <none>
I20260810 19:21:38.026682 26927 fs_manager.cc:359] Metadata directory not provided
I20260810 19:21:38.026886 26927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:21:38.027367 26927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:21:38 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389698, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:21:38.045495 26927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "ae9eaaf6614b45259afb446f53efd61a"
format_stamp: "Formatted at 2026-08-10 19:21:38 on dist-test-slave-7lbf"
server_key: "f6bd17d5256978dffe17c0b487da8aa5"
server_key_iv: "222e6005697954d9ece66bd68f493ae0"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:38.046929 26927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "ae9eaaf6614b45259afb446f53efd61a"
format_stamp: "Formatted at 2026-08-10 19:21:38 on dist-test-slave-7lbf"
server_key: "f6bd17d5256978dffe17c0b487da8aa5"
server_key_iv: "222e6005697954d9ece66bd68f493ae0"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:38.055092 26927 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.006s
I20260810 19:21:38.075018 26959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:38.076851 26927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260810 19:21:38.077087 26927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "ae9eaaf6614b45259afb446f53efd61a"
format_stamp: "Formatted at 2026-08-10 19:21:38 on dist-test-slave-7lbf"
server_key: "f6bd17d5256978dffe17c0b487da8aa5"
server_key_iv: "222e6005697954d9ece66bd68f493ae0"
server_key_version: "kuduclusterkey@0"
I20260810 19:21:38.077399 26927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:21:38.099351 26927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:21:38.106015 26927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:21:38.106616 26927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:21:38.108670 26927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:21:38.111563 26927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:21:38.111728 26927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:38.112006 26927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:21:38.112118 26927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:21:38.192701 26927 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:39299
I20260810 19:21:38.192843 27156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:39299 every 8 connection(s)
I20260810 19:21:38.194804 26927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:21:38.200623 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 26927
I20260810 19:21:38.200791 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:21:38.201431 7340 external_mini_cluster.cc:1428] Setting key dc973dff0f4352f5d43dea9eadf0a08f
Aug 10 19:21:38 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389698, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:21:38.222401 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.198307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:47829 (local address 127.7.43.62:42657)
0810 19:21:38.199199 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.199204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.200468 (+ 1264us) server_negotiation.cc:407] Connection header received
0810 19:21:38.202672 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.202677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.202765 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.202916 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:38.205436 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.206581 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.208729 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.209434 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.209584 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.214510 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:38.214516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:38.214520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:38.214555 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:38.218883 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:38.219754 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:38.219759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:38.219761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:38.219956 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:38.220370 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:38.220373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:38.220375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:38.220846 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:38.220979 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.221766 (+ 787us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.221975 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":552,"threads_started":1}
I20260810 19:21:38.223675 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.198991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42657 (local address 127.7.43.3:47829)
0810 19:21:38.200248 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.200314 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.202366 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.203193 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.203205 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.204079 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:21:38.205245 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.205270 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.206757 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.206761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.208528 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.208569 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.209794 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.209803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.210171 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.210972 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:21:38.210999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:21:38.214156 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:21:38.219072 (+ 4916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:21:38.219083 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:21:38.219103 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:21:38.219620 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:21:38.220054 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:21:38.220057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:21:38.220059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:21:38.220281 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:21:38.220988 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:21:38.220996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:21:38.221336 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.221676 (+ 340us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.222213 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":638,"threads_started":1}
I20260810 19:21:38.225589 27158 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42657
I20260810 19:21:38.225935 27158 heartbeater.cc:464] Registering TS with master...
I20260810 19:21:38.226657 27158 heartbeater.cc:511] Master 127.7.43.62:42657 requested a full tablet report, sending...
I20260810 19:21:38.228443 26305 ts_manager.cc:194] Registered new tserver with Master: ae9eaaf6614b45259afb446f53efd61a (127.7.43.3:39299)
I20260810 19:21:38.229674 26305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:47829
I20260810 19:21:38.232870 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:21:38.251757 26918 heartbeater.cc:503] Master 127.7.43.62:42657 was elected leader, sending a full tablet report...
I20260810 19:21:38.258610 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.239196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.7.43.62:42657)
0810 19:21:38.239418 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.239425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.240336 (+ 911us) server_negotiation.cc:407] Connection header received
0810 19:21:38.240514 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.240518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.240601 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.240686 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:38.243332 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.244386 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.246473 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.247298 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.247413 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.250883 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:38.250889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:38.250893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:38.250959 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:38.255060 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:38.255961 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:38.255965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:38.255967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:38.256147 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:38.256566 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:38.256571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:38.256574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:38.256956 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:38.257212 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.257753 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.258090 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:21:38.263881 26305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:21:38.276152 26305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:21:38.322243 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.295177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.7.43.3:39299)
0810 19:21:38.306224 (+ 11047us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.306236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.308920 (+ 2684us) server_negotiation.cc:407] Connection header received
0810 19:21:38.309379 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.309389 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.309782 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.310070 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.312036 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.314263 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.318737 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.320377 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.320565 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.320913 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.321298 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.321567 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10769,"thread_start_us":11091,"threads_started":1}
I20260810 19:21:38.323107 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.295221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.7.43.2:41633)
0810 19:21:38.307427 (+ 12206us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.307438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.310489 (+ 3051us) server_negotiation.cc:407] Connection header received
0810 19:21:38.310672 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.310682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.310945 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.311226 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.313204 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.315355 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.318873 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.321072 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.321218 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.321584 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.322319 (+ 735us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.322543 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12004,"thread_start_us":11685,"threads_started":1}
I20260810 19:21:38.323948 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.294459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39299 (local address 127.0.0.1:51388)
0810 19:21:38.308696 (+ 14237us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.308741 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.309072 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.310129 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.310136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.310166 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.311487 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.311507 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.314399 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.314405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.318537 (+ 4132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.318556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.320677 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.320685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.321014 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.321060 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.322821 (+ 1761us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.323020 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13904,"mutex_wait_us":31,"thread_start_us":15237,"threads_started":1}
I20260810 19:21:38.324384 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.294665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:41633 (local address 127.0.0.1:35892)
0810 19:21:38.309762 (+ 15097us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.309783 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.309943 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.311196 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.311201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.311227 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.312983 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.312998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.315549 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.315552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.318695 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.318711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.321832 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.321841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.322073 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.322116 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.323407 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.323558 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14934,"mutex_wait_us":109,"thread_start_us":13408,"threads_started":1}
I20260810 19:21:38.325397 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.299260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34127 (local address 127.0.0.1:60436)
0810 19:21:38.312021 (+ 12761us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.312072 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.312244 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.313427 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.313432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.313471 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.314381 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.314397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.317087 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.317097 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.321122 (+ 4025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.321137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.324336 (+ 3199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.324344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.324591 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.324625 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.324718 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.324855 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12568,"thread_start_us":11775,"threads_started":1}
I20260810 19:21:38.329716 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.300309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.7.43.1:34127)
0810 19:21:38.309673 (+ 9364us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.309682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.312205 (+ 2523us) server_negotiation.cc:407] Connection header received
0810 19:21:38.312421 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.312427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.312744 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.312985 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.314696 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.316855 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.321321 (+ 4466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.323555 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.323664 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.323984 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.324851 (+ 867us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.325103 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9102,"thread_start_us":8270,"threads_started":1}
I20260810 19:21:38.329661 26821 tablet_service.cc:1467] Processing CreateTablet for tablet 9960011a035941e8b7be0c0808998016 (DEFAULT_TABLE table=test-table [id=6d9a05433370410d858ff655634e4698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:21:38.329723 27039 tablet_service.cc:1467] Processing CreateTablet for tablet 9960011a035941e8b7be0c0808998016 (DEFAULT_TABLE table=test-table [id=6d9a05433370410d858ff655634e4698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:21:38.332065 26821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9960011a035941e8b7be0c0808998016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:21:38.332525 27039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9960011a035941e8b7be0c0808998016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:21:38.334204 26561 tablet_service.cc:1467] Processing CreateTablet for tablet 9960011a035941e8b7be0c0808998016 (DEFAULT_TABLE table=test-table [id=6d9a05433370410d858ff655634e4698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:21:38.336130 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9960011a035941e8b7be0c0808998016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:21:38.353549 27204 tablet_bootstrap.cc:492] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Bootstrap starting.
I20260810 19:21:38.359623 27204 tablet_bootstrap.cc:654] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Neither blocks nor log segments found. Creating new log.
I20260810 19:21:38.359884 27206 tablet_bootstrap.cc:492] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Bootstrap starting.
I20260810 19:21:38.362037 27204 log.cc:824] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Log is configured to *not* fsync() on all Append() calls
I20260810 19:21:38.363457 27207 tablet_bootstrap.cc:492] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Bootstrap starting.
I20260810 19:21:38.368868 27206 tablet_bootstrap.cc:654] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Neither blocks nor log segments found. Creating new log.
I20260810 19:21:38.368906 27207 tablet_bootstrap.cc:654] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Neither blocks nor log segments found. Creating new log.
I20260810 19:21:38.370913 27207 log.cc:824] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Log is configured to *not* fsync() on all Append() calls
I20260810 19:21:38.371361 27206 log.cc:824] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:21:38.371908 27204 tablet_bootstrap.cc:492] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: No bootstrap required, opened a new log
I20260810 19:21:38.372396 27204 ts_tablet_manager.cc:1397] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260810 19:21:38.375856 27207 tablet_bootstrap.cc:492] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: No bootstrap required, opened a new log
I20260810 19:21:38.376784 27207 ts_tablet_manager.cc:1397] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260810 19:21:38.377961 27206 tablet_bootstrap.cc:492] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: No bootstrap required, opened a new log
I20260810 19:21:38.378388 27206 ts_tablet_manager.cc:1397] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260810 19:21:38.388204 27204 raft_consensus.cc:348] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.388967 27204 raft_consensus.cc:374] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:21:38.389149 27204 raft_consensus.cc:729] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad147ae992664ac4a1120512c60c6740, State: Initialized, Role: FOLLOWER
I20260810 19:21:38.389101 27207 raft_consensus.cc:348] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.389762 27207 raft_consensus.cc:374] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:21:38.389969 27207 raft_consensus.cc:729] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44ca59ce3014de0bc2e7b80fa1d61ce, State: Initialized, Role: FOLLOWER
I20260810 19:21:38.390077 27204 consensus_queue.cc:260] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.390733 27207 consensus_queue.cc:260] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.394861 27206 raft_consensus.cc:348] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.395421 27204 ts_tablet_manager.cc:1428] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Time spent starting tablet: real 0.023s user 0.011s sys 0.011s
I20260810 19:21:38.395543 27206 raft_consensus.cc:374] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:21:38.395781 27206 raft_consensus.cc:729] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9eaaf6614b45259afb446f53efd61a, State: Initialized, Role: FOLLOWER
I20260810 19:21:38.396915 27206 consensus_queue.cc:260] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.398552 27207 ts_tablet_manager.cc:1428] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Time spent starting tablet: real 0.022s user 0.017s sys 0.003s
I20260810 19:21:38.401784 27158 heartbeater.cc:503] Master 127.7.43.62:42657 was elected leader, sending a full tablet report...
I20260810 19:21:38.402694 27206 ts_tablet_manager.cc:1428] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Time spent starting tablet: real 0.024s user 0.012s sys 0.012s
W20260810 19:21:38.447691 27163 tablet.cc:2369] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:21:38.459671 26920 tablet.cc:2369] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:21:38.580060 26683 tablet.cc:2369] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:21:38.671236 27221 raft_consensus.cc:482] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:21:38.671737 27221 raft_consensus.cc:504] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.674170 27221 leader_election.cc:288] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae9eaaf6614b45259afb446f53efd61a (127.7.43.3:39299), a44ca59ce3014de0bc2e7b80fa1d61ce (127.7.43.2:41633)
I20260810 19:21:38.675706 27224 raft_consensus.cc:482] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:21:38.676316 27224 raft_consensus.cc:504] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.681838 27224 leader_election.cc:288] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae9eaaf6614b45259afb446f53efd61a (127.7.43.3:39299), ad147ae992664ac4a1120512c60c6740 (127.7.43.1:34127)
I20260810 19:21:38.690420 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.675470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39299 (local address 127.7.43.1:54999)
0810 19:21:38.677382 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.677426 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.677724 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.678135 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.678139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.678166 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.680766 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.680783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.682860 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.682866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.686381 (+ 3515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.686399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.689098 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.689107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.689362 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.689402 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.689536 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.689705 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"spinlock_wait_cycles":62720,"thread_start_us":1337,"threads_started":1}
I20260810 19:21:38.691383 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.674477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:54999 (local address 127.7.43.3:39299)
0810 19:21:38.674706 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.674713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.677605 (+ 2892us) server_negotiation.cc:407] Connection header received
0810 19:21:38.677899 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.677905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.677999 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.678095 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.680900 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.682706 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.686635 (+ 3929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.688526 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.688628 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.688779 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.690839 (+ 2060us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.691006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:21:38.691816 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.674837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:50891 (local address 127.7.43.2:41633)
0810 19:21:38.675144 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.675149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.677672 (+ 2523us) server_negotiation.cc:407] Connection header received
0810 19:21:38.677942 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.677947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.678033 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.678094 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.681386 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.683214 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.688434 (+ 5220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.690343 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.690507 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.690661 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.691393 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.691538 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":69}
I20260810 19:21:38.692612 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.675533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:41633 (local address 127.7.43.1:50891)
0810 19:21:38.677555 (+ 2022us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.677574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.677753 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.678354 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.678357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.678391 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.681069 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.681083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.684726 (+ 3643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.684732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.688154 (+ 3422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.688171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.690980 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.690988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.691188 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.691226 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.692101 (+ 875us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.692260 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1813,"thread_start_us":1681,"threads_started":1}
I20260810 19:21:38.693819 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.679475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39299 (local address 127.7.43.2:47065)
0810 19:21:38.680739 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.680758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.680917 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.681416 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.681422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.681466 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.684228 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.684244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.686605 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.686615 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.689954 (+ 3339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.689971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.692813 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.692821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.693009 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.693045 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.693136 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.693277 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":28,"thread_start_us":1555,"threads_started":1}
I20260810 19:21:38.693682 27080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "ad147ae992664ac4a1120512c60c6740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9eaaf6614b45259afb446f53efd61a" is_pre_election: true
I20260810 19:21:38.694638 27080 raft_consensus.cc:2454] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad147ae992664ac4a1120512c60c6740 in term 0.
I20260810 19:21:38.694291 26847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "ad147ae992664ac4a1120512c60c6740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" is_pre_election: true
I20260810 19:21:38.695143 26847 raft_consensus.cc:2454] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad147ae992664ac4a1120512c60c6740 in term 0.
I20260810 19:21:38.695684 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.679775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:47065 (local address 127.7.43.3:39299)
0810 19:21:38.680734 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.680741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.680821 (+ 80us) server_negotiation.cc:407] Connection header received
0810 19:21:38.681119 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.681124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.681210 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.681310 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.684494 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.686414 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.690196 (+ 3782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.692243 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.692342 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.692489 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.693209 (+ 720us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.693390 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":579,"threads_started":1}
I20260810 19:21:38.695626 26488 leader_election.cc:302] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad147ae992664ac4a1120512c60c6740, ae9eaaf6614b45259afb446f53efd61a; no voters:
I20260810 19:21:38.696856 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.680784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:53629 (local address 127.7.43.1:34127)
0810 19:21:38.681036 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.681042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.684886 (+ 3844us) server_negotiation.cc:407] Connection header received
0810 19:21:38.685176 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.685182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.685253 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.685309 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:21:38.686783 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.688526 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.692582 (+ 4056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.695037 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.695151 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.695242 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.696357 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.696510 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260810 19:21:38.696972 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.680289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34127 (local address 127.7.43.2:53629)
0810 19:21:38.684785 (+ 4496us) negotiation.cc:107] Waiting for socket to connect
0810 19:21:38.684806 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:21:38.685021 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:21:38.685594 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:21:38.685598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:21:38.685623 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:21:38.686554 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.686569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.688743 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.688753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.692358 (+ 3605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:21:38.692374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.695629 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.695638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:21:38.695863 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.695903 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:21:38.696128 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 19:21:38.696292 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4324,"thread_start_us":4051,"threads_started":1}
I20260810 19:21:38.696904 27221 raft_consensus.cc:2781] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:21:38.697106 27080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9eaaf6614b45259afb446f53efd61a" is_pre_election: true
I20260810 19:21:38.697187 27221 raft_consensus.cc:482] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:21:38.697391 27080 raft_consensus.cc:2454] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44ca59ce3014de0bc2e7b80fa1d61ce in term 0.
I20260810 19:21:38.697450 27221 raft_consensus.cc:3037] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:21:38.698004 26738 leader_election.cc:302] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a44ca59ce3014de0bc2e7b80fa1d61ce, ae9eaaf6614b45259afb446f53efd61a; no voters:
I20260810 19:21:38.698463 27224 raft_consensus.cc:2781] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:21:38.698668 27224 raft_consensus.cc:482] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:21:38.698877 27224 raft_consensus.cc:3037] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:21:38.698563 26604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad147ae992664ac4a1120512c60c6740" is_pre_election: true
I20260810 19:21:38.701989 27221 raft_consensus.cc:504] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.702384 27224 raft_consensus.cc:504] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.702819 26604 raft_consensus.cc:2379] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a44ca59ce3014de0bc2e7b80fa1d61ce in current term 1: Already voted for candidate ad147ae992664ac4a1120512c60c6740 in this term.
I20260810 19:21:38.703472 27221 leader_election.cc:288] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [CANDIDATE]: Term 1 election: Requested vote from peers ae9eaaf6614b45259afb446f53efd61a (127.7.43.3:39299), a44ca59ce3014de0bc2e7b80fa1d61ce (127.7.43.2:41633)
I20260810 19:21:38.704195 27080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "ad147ae992664ac4a1120512c60c6740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9eaaf6614b45259afb446f53efd61a"
I20260810 19:21:38.704216 26847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "ad147ae992664ac4a1120512c60c6740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce"
I20260810 19:21:38.704041 27078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9eaaf6614b45259afb446f53efd61a"
I20260810 19:21:38.704488 27080 raft_consensus.cc:3037] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:21:38.704592 26847 raft_consensus.cc:2379] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ad147ae992664ac4a1120512c60c6740 in current term 1: Already voted for candidate a44ca59ce3014de0bc2e7b80fa1d61ce in this term.
I20260810 19:21:38.708097 27224 leader_election.cc:288] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [CANDIDATE]: Term 1 election: Requested vote from peers ae9eaaf6614b45259afb446f53efd61a (127.7.43.3:39299), ad147ae992664ac4a1120512c60c6740 (127.7.43.1:34127)
I20260810 19:21:38.708529 27080 raft_consensus.cc:2454] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad147ae992664ac4a1120512c60c6740 in term 1.
I20260810 19:21:38.708902 26604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9960011a035941e8b7be0c0808998016" candidate_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad147ae992664ac4a1120512c60c6740"
I20260810 19:21:38.709024 26488 leader_election.cc:302] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad147ae992664ac4a1120512c60c6740, ae9eaaf6614b45259afb446f53efd61a; no voters: a44ca59ce3014de0bc2e7b80fa1d61ce
I20260810 19:21:38.709278 26604 raft_consensus.cc:2379] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a44ca59ce3014de0bc2e7b80fa1d61ce in current term 1: Already voted for candidate ad147ae992664ac4a1120512c60c6740 in this term.
I20260810 19:21:38.709587 27221 raft_consensus.cc:2781] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:21:38.709837 26736 leader_election.cc:302] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44ca59ce3014de0bc2e7b80fa1d61ce; no voters: ad147ae992664ac4a1120512c60c6740, ae9eaaf6614b45259afb446f53efd61a
I20260810 19:21:38.710351 27224 raft_consensus.cc:2726] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 19:21:38.710484 27221 raft_consensus.cc:686] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 LEADER]: Becoming Leader. State: Replica: ad147ae992664ac4a1120512c60c6740, State: Running, Role: LEADER
I20260810 19:21:38.711499 27221 consensus_queue.cc:237] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } }
I20260810 19:21:38.717274 26305 catalog_manager.cc:5495] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad147ae992664ac4a1120512c60c6740 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "ad147ae992664ac4a1120512c60c6740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad147ae992664ac4a1120512c60c6740" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34127 } health_report { overall_health: HEALTHY } } }
I20260810 19:21:38.788556 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.781103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.7.43.1:34127)
0810 19:21:38.781264 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.781268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.782163 (+ 895us) server_negotiation.cc:407] Connection header received
0810 19:21:38.782377 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.782384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.782516 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.782678 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:21:38.783038 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.783960 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.785627 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.786313 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.786452 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.787123 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:21:38.787854 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:21:38.788002 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.788129 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.788226 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:21:38.799104 27080 raft_consensus.cc:1261] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 1 FOLLOWER]: Refusing update from remote peer ad147ae992664ac4a1120512c60c6740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:21:38.799271 26847 raft_consensus.cc:1261] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 1 FOLLOWER]: Refusing update from remote peer ad147ae992664ac4a1120512c60c6740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:21:38.800355 27239 consensus_queue.cc:1035] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9eaaf6614b45259afb446f53efd61a" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:21:38.800799 27221 consensus_queue.cc:1035] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44ca59ce3014de0bc2e7b80fa1d61ce" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:21:38.836000 26305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:21:38.836347 26305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:21:38.840723 26305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d799ebc0f7b46229e7047543213971f: Sending DeleteTablet for 3 replicas of tablet 9960011a035941e8b7be0c0808998016
I20260810 19:21:38.842126 26821 tablet_service.cc:1514] Processing DeleteTablet for tablet 9960011a035941e8b7be0c0808998016 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:21:38 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:35892
I20260810 19:21:38.842218 27039 tablet_service.cc:1514] Processing DeleteTablet for tablet 9960011a035941e8b7be0c0808998016 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:21:38 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:51388
I20260810 19:21:38.843250 26561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9960011a035941e8b7be0c0808998016 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:21:38 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:60436
I20260810 19:21:38.843736 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 26453
I20260810 19:21:38.854077 27268 tablet_replica.cc:330] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: stopping tablet replica
I20260810 19:21:38.854533 27264 tablet_replica.cc:330] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: stopping tablet replica
I20260810 19:21:38.854866 27268 raft_consensus.cc:2229] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:21:38.855087 27264 raft_consensus.cc:2229] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:21:38.855886 27268 raft_consensus.cc:2258] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:21:38.855962 27264 raft_consensus.cc:2258] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:21:38.857555 27266 tablet_replica.cc:330] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: stopping tablet replica
I20260810 19:21:38.858242 27266 raft_consensus.cc:2229] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:21:38.858611 27264 ts_tablet_manager.cc:1905] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:21:38.858618 27268 ts_tablet_manager.cc:1905] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:21:38.859140 27266 raft_consensus.cc:2258] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:21:38.862111 27266 ts_tablet_manager.cc:1905] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:21:38.867939 27264 ts_tablet_manager.cc:1918] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:21:38.868351 27264 log.cc:1196] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/9960011a035941e8b7be0c0808998016
I20260810 19:21:38.868613 27268 ts_tablet_manager.cc:1918] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:21:38.868985 27268 log.cc:1196] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/9960011a035941e8b7be0c0808998016
I20260810 19:21:38.869351 27264 ts_tablet_manager.cc:1939] T 9960011a035941e8b7be0c0808998016 P a44ca59ce3014de0bc2e7b80fa1d61ce: Deleting consensus metadata
I20260810 19:21:38.869961 27268 ts_tablet_manager.cc:1939] T 9960011a035941e8b7be0c0808998016 P ad147ae992664ac4a1120512c60c6740: Deleting consensus metadata
I20260810 19:21:38.872135 26283 catalog_manager.cc:4841] TS a44ca59ce3014de0bc2e7b80fa1d61ce (127.7.43.2:41633): tablet 9960011a035941e8b7be0c0808998016 (table test-table [id=6d9a05433370410d858ff655634e4698]) successfully deleted
I20260810 19:21:38.872887 26279 catalog_manager.cc:4841] TS ad147ae992664ac4a1120512c60c6740 (127.7.43.1:34127): tablet 9960011a035941e8b7be0c0808998016 (table test-table [id=6d9a05433370410d858ff655634e4698]) successfully deleted
I20260810 19:21:38.877463 27266 ts_tablet_manager.cc:1918] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:21:38.877731 27266 log.cc:1196] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/9960011a035941e8b7be0c0808998016
I20260810 19:21:38.878361 27266 ts_tablet_manager.cc:1939] T 9960011a035941e8b7be0c0808998016 P ae9eaaf6614b45259afb446f53efd61a: Deleting consensus metadata
I20260810 19:21:38.880199 26282 catalog_manager.cc:4841] TS ae9eaaf6614b45259afb446f53efd61a (127.7.43.3:39299): tablet 9960011a035941e8b7be0c0808998016 (table test-table [id=6d9a05433370410d858ff655634e4698]) successfully deleted
Aug 10 19:21:38 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:21:38.892277 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:38.844899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.7.43.1:34127)
0810 19:21:38.845117 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:38.845123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:38.860372 (+ 15249us) server_negotiation.cc:407] Connection header received
0810 19:21:38.864565 (+ 4193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:38.864572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:38.864659 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:38.864737 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:38.871159 (+ 6422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.872612 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.875106 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:38.875947 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:38.876094 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:38.883118 (+ 7024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:38.883135 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:38.883152 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:38.883217 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:38.888172 (+ 4955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:38.889368 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:38.889376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:38.889379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:38.889637 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:38.890325 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:38.890329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:38.890331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:38.890952 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:38.891332 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:38.891646 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 19:21:38.891863 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:21:38.893091 26676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:21:38.995716 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 26453
I20260810 19:21:39.020522 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 26696
Aug 10 19:21:39 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:21:39.040259 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:39.021275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.7.43.2:41633)
0810 19:21:39.021462 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:39.021467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:39.021485 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:21:39.021733 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:39.021737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:39.021816 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:39.021899 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:39.023245 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:39.023981 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:39.025258 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:39.025693 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:39.025789 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:39.034429 (+ 8640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:39.034447 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:39.034466 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:39.034509 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:39.037313 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:39.038169 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:39.038178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:39.038180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:39.038316 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:39.038697 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:39.038702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:39.038704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:39.039154 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:39.039392 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:39.039739 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 19:21:39.039919 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:21:39.041031 26911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:21:39.133226 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 26696
I20260810 19:21:39.164425 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 26927
Aug 10 19:21:39 dist-test-slave-7lbf krb5kdc[25454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:21:39.179713 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:21:39.165234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.7.43.3:39299)
0810 19:21:39.165457 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 19:21:39.165463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:21:39.165486 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:21:39.165647 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:21:39.165653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:21:39.165741 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:21:39.165868 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:21:39.167273 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:39.167998 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:39.169444 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:21:39.169896 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:21:39.169992 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:21:39.174379 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:21:39.174398 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:21:39.174416 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:21:39.174470 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:21:39.177158 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:39.177881 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:39.177888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:39.177890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:39.178048 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:21:39.178359 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:21:39.178362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:21:39.178364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:21:39.178695 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:21:39.178910 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 19:21:39.179262 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 19:21:39.179462 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 19:21:39.180321 27149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:21:39.270481 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 26927
I20260810 19:21:39.295997 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 26250
I20260810 19:21:39.297237 26362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:21:39.409767 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 26250
I20260810 19:21:39.444968 7340 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 19:21:39.790589 7340 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 19:21:39.790896 7340 mini_ranger.cc:66] Stopping Ranger...
I20260810 19:21:40.136868 7340 mini_ranger.cc:68] Stopped Ranger
2026-08-10 19:21:40.137 UTC [25509] LOG: received smart shutdown request
2026-08-10 19:21:40.140 UTC [25509] LOG: background worker "logical replication launcher" (PID 25530) exited with exit code 1
2026-08-10 19:21:40.141 UTC [25522] LOG: shutting down
2026-08-10 19:21:40.292 UTC [25509] LOG: database system is shut down
2026-08-10T19:21:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52456 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:21:40 dist-test-slave-7lbf krb5kdc[27294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:21:40 dist-test-slave-7lbf krb5kdc[27294](info): set up 2 sockets
Aug 10 19:21:40 dist-test-slave-7lbf krb5kdc[27294](info): commencing operation
krb5kdc: starting...
W20260810 19:21:41.665262 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:21:41 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:21:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/rangeradmin_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/rangeradmin_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/rangerlookup_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/rangerlookup_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
I20260810 19:21:42.088510 7340 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuOTVHW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/postgres -l logfile start
2026-08-10 19:21:44.496 UTC [27342] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 19:21:44.496 UTC [27342] LOG: listening on IPv4 address "127.7.43.20", port 58591
2026-08-10 19:21:44.499 UTC [27342] LOG: listening on Unix socket "/tmp/.s.PGSQL.58591"
2026-08-10 19:21:44.522 UTC [27344] LOG: database system was shut down at 2026-08-10 19:21:43 UTC
2026-08-10 19:21:44.525 UTC [27342] LOG: database system is ready to accept connections
I20260810 19:21:45.642546 7340 mini_postgres.cc:96] Postgres bound to 58591
2026-08-10 19:21:45.671 UTC [27359] FATAL: database "slave" does not exist
127.7.43.20:58591 - accepting connections
I20260810 19:21:45.671702 7340 mini_ranger.cc:161] Starting Ranger...
I20260810 19:21:45.706717 7340 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 19:21:46.179487 7340 mini_ranger.cc:87] Created ranger Postgres database
I20260810 19:21:46.179615 7340 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-admin
2026-08-10 19:21:46,462 [I] DB FLAVOR :POSTGRES
2026-08-10 19:21:46,462 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:21:46,462 [I] Checking connection..
2026-08-10 19:21:46,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 19:21:46,699 [I] Checking connection passed.
2026-08-10 19:21:46,700 [I] --------- Verifying version history table ---------
2026-08-10 19:21:46,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:21:46,941 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:21:46,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:21:47,209 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:21:47,209 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 19:21:47,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 19:21:47.471 UTC [27467] WARNING: there is no transaction in progress
2026-08-10 19:21:47,480 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 19:21:47,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:21:47,767 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 19:21:47,767 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:21:47,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 19:21:48,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 19:21:48,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7lbf.c.gcp-upstream.internal','N') ;"
2026-08-10 19:21:48,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:21:48,771 [I] Table x_portal_user does not exist in database ranger
2026-08-10 19:21:48,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:21:49,024 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 19:21:49,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:21:49,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:21:49,543 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 19:21:49,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 19:21:50.495 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.511 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.527 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.542 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.558 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.573 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.626 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.632 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.645 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.655 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.666 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.680 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.689 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.695 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.701 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:50.707 UTC [27641] WARNING: there is no transaction in progress
2026-08-10 19:21:51,297 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 19:21:51,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 19:21:51,553 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 19:21:51,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:21:51,818 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 19:21:51,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:21:52,072 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 19:21:52,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:21:52,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:21:52,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7lbf.c.gcp-upstream.internal';"
2026-08-10 19:21:52,838 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 19:21:52,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 19:21:53,098 [I] DB_PATCHES have already been applied
I20260810 19:21:53.103891 7340 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuOTVHW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 19:21:53.103977 7340 mini_ranger.cc:193] Using host: 127.7.43.20
I20260810 19:21:53.125239 7340 mini_ranger.cc:227] Ranger admin URL: http://127.7.43.20:43687
Aug 10, 2026 7:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:21:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:21:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 19:21:54.453935 7340 mini_ranger.cc:160] Time spent starting Ranger: real 8.782s user 0.004s sys 0.065s
Aug 10, 2026 7:22:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:22:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:22:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:22:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:22:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 19:22:16.078517 7340 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.43.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.43.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/rangerkms_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.43.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/rangerkms_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.43.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.43.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.43.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/HTTP_127.7.43.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 19:22:16 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389736, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 19:22:16.293345 7340 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 19:22:16.320261 7340 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 19:22:17.123778 7340 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 19:22:17.123915 7340 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms
I20260810 19:22:17.123966 7340 mini_ranger_kms.cc:227] Using postgres at 127.7.43.20:58591
2026-08-10 19:22:17,487 [I] DB FLAVOR :POSTGRES
2026-08-10 19:22:17,487 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:22:17,487 [I] Checking connection
2026-08-10 19:22:17,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 19:22:17,739 [I] connection success
2026-08-10 19:22:17,740 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 19:22:17,740 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 19:22:17,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 19:22:17,981 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 19:22:17,982 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:22:17,982 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 19:22:17,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.43.20:58591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 19:22:18,234 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 19:22:18.703311 7340 mini_ranger_kms.cc:310] Created kms service
I20260810 19:22:18.871529 7340 mini_ranger_kms.cc:326] Created kudu user
I20260810 19:22:18.917407 7340 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 19:22:18 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389736, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:19.320459 7340 mini_ranger_kms.cc:384] Added ranger policy
I20260810 19:22:19.320828 7340 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuOTVHW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuOTVHW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 19:22:19.320899 7340 mini_ranger_kms.cc:240] Using host: 127.7.43.20
I20260810 19:22:19.342989 7340 mini_ranger_kms.cc:276] Ranger KMS PID: 27932
I20260810 19:22:19.343133 7340 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.43.20:41763
Aug 10, 2026 7:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 19:22:21 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786389741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.43.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:22.005851 7340 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.712s user 0.082s sys 0.323s
I20260810 19:22:22.006109 7340 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 19:22:22.006290 7340 mini_ranger_kms.cc:401] 127.7.43.20:41763/kms/v1/keys
Aug 10 19:22:22 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.43.20@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
Aug 10 19:22:22 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.43.20@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
Aug 10 19:22:24 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:22:24.917476 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:46127
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:46127
--ranger_config_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:25.268121 28005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:25.268438 28005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:25.268563 28005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:25.275763 28005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:25.275888 28005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:25.275961 28005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:25.276041 28005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:25.276108 28005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:25.286173 28005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.43.20:41763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:46127
--ranger_config_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:25.287473 28005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:25.288753 28005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:25.300166 28012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:25.300115 28016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:25.300204 28013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:25.301100 28005 server_base.cc:1034] running on GCE node
I20260810 19:22:25.301899 28005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:25.303264 28005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:25.304488 28005 hybrid_clock.cc:648] HybridClock initialized: now 1786389745304448 us; error 31 us; skew 500 ppm
I20260810 19:22:25.304841 28005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:25 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:25.307849 28005 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:25.309197 28005 webserver.cc:460] Webserver started at http://127.7.43.62:40979/ using document root <none> and password file <none>
I20260810 19:22:25.309912 28005 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:25.310026 28005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:25.310336 28005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:22:25 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389745, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:25.367904 28005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "11c2ba66ddfc486fafb46f8300af70d4"
format_stamp: "Formatted at 2026-08-10 19:22:25 on dist-test-slave-7lbf"
server_key: "cbdf02f51e23817c4cc2c1b1ac8d70b1"
server_key_iv: "b0d55a5cbcb83e6b04c537384c9a1419"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:25.369035 28005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "11c2ba66ddfc486fafb46f8300af70d4"
format_stamp: "Formatted at 2026-08-10 19:22:25 on dist-test-slave-7lbf"
server_key: "cbdf02f51e23817c4cc2c1b1ac8d70b1"
server_key_iv: "b0d55a5cbcb83e6b04c537384c9a1419"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:25.376209 28005 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260810 19:22:25.396991 28027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:25.398952 28005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:22:25.399168 28005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "11c2ba66ddfc486fafb46f8300af70d4"
format_stamp: "Formatted at 2026-08-10 19:22:25 on dist-test-slave-7lbf"
server_key: "cbdf02f51e23817c4cc2c1b1ac8d70b1"
server_key_iv: "b0d55a5cbcb83e6b04c537384c9a1419"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:25.399443 28005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:25.429920 28005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:25.433947 28005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:25.434249 28005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:25.451589 28005 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:46127
I20260810 19:22:25.451608 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:46127 every 8 connection(s)
I20260810 19:22:25.452925 28005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:25.456693 28119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:25.457844 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 28005
I20260810 19:22:25.458083 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:22:25.458986 7340 external_mini_cluster.cc:1428] Setting key e1f528df3409ab5666e8eb9b86a75a9b
I20260810 19:22:25.471191 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Bootstrap starting.
Aug 10 19:22:25 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:25.474506 28119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:25.475741 28119 log.cc:824] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:25.478686 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: No bootstrap required, opened a new log
I20260810 19:22:25.481191 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:25.461097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.7.43.62:46127)
0810 19:22:25.461961 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:25.461976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:25.462027 (+ 51us) server_negotiation.cc:407] Connection header received
0810 19:22:25.463625 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:25.463648 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:25.464058 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:25.464384 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:25.466023 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:25.467356 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:25.469208 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:25.470219 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:25.470403 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:25.474372 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:25.474380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:25.474398 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:25.474434 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:25.477343 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:25.478017 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:25.478024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:25.478030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:25.478192 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:25.478564 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:25.478567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:25.478569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:25.479081 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:25.479365 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:25.479803 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 19:22:25.480168 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":411,"threads_started":1}
I20260810 19:22:25.486438 28119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:25.486779 28119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:25.486873 28119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c2ba66ddfc486fafb46f8300af70d4, State: Initialized, Role: FOLLOWER
I20260810 19:22:25.487449 28119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:25.487735 28119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:22:25.487879 28119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:22:25.488050 28119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:25.490226 28119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:25.490728 28119 leader_election.cc:302] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c2ba66ddfc486fafb46f8300af70d4; no voters:
I20260810 19:22:25.491245 28119 leader_election.cc:288] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:22:25.491379 28127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:25.492007 28127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 LEADER]: Becoming Leader. State: Replica: 11c2ba66ddfc486fafb46f8300af70d4, State: Running, Role: LEADER
I20260810 19:22:25.492508 28127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:25.493222 28119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:25.496855 28129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c2ba66ddfc486fafb46f8300af70d4. Latest consensus state: current_term: 1 leader_uuid: "11c2ba66ddfc486fafb46f8300af70d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } } }
I20260810 19:22:25.497277 28129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:25.497757 28128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c2ba66ddfc486fafb46f8300af70d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } } }
I20260810 19:22:25.498041 28128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:25.500530 28119 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:22:25.501137 28139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:25.514865 28139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:25.527384 28139 catalog_manager.cc:1347] Generated new cluster ID: 05ea8796048d471c8fc19546eb8adf09
I20260810 19:22:25.527524 28139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:22:25.548700 28139 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:22:25.550174 28139 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:22:25.563483 28139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Generated new TSK 0
I20260810 19:22:25.564291 28139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:22:26.317052 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:46127
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:22:26 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786389746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 19:22:26.751101 28192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:26.751542 28192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:26.751760 28192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:26.763904 28192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:26.764081 28192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:26.764263 28192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:22:26.780174 28192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.43.20:41763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:26.782147 28192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:26.783944 28192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:26.811838 28214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:26.811774 28215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:26.811964 28218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:26.812918 28192 server_base.cc:1034] running on GCE node
I20260810 19:22:26.813654 28192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:26.814726 28192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:26.815932 28192 hybrid_clock.cc:648] HybridClock initialized: now 1786389746815855 us; error 71 us; skew 500 ppm
I20260810 19:22:26.816268 28192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:26 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:26.819645 28192 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:22:26.821245 28192 webserver.cc:460] Webserver started at http://127.7.43.1:40999/ using document root <none> and password file <none>
I20260810 19:22:26.821995 28192 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:26.822116 28192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:26.822420 28192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:22:26 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:26.852370 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "3d87ae5782294699b382c4c659c3fdb6"
format_stamp: "Formatted at 2026-08-10 19:22:26 on dist-test-slave-7lbf"
server_key: "1e20efecc54b335e2d125bd3c55e37fc"
server_key_iv: "1e99b5d5cbf30d93f8ad666a3314f6cb"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:26.853725 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "3d87ae5782294699b382c4c659c3fdb6"
format_stamp: "Formatted at 2026-08-10 19:22:26 on dist-test-slave-7lbf"
server_key: "1e20efecc54b335e2d125bd3c55e37fc"
server_key_iv: "1e99b5d5cbf30d93f8ad666a3314f6cb"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:26.868398 28192 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.008s
I20260810 19:22:26.888638 28230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:26.890771 28192 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260810 19:22:26.891077 28192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "3d87ae5782294699b382c4c659c3fdb6"
format_stamp: "Formatted at 2026-08-10 19:22:26 on dist-test-slave-7lbf"
server_key: "1e20efecc54b335e2d125bd3c55e37fc"
server_key_iv: "1e99b5d5cbf30d93f8ad666a3314f6cb"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:26.891444 28192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:26.986955 28192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:26.993306 28192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:26.993747 28192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:27.001224 28192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:27.009690 28192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:27.009912 28192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:27.010133 28192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:27.010257 28192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 19:22:27 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:27.272359 28192 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:37707
I20260810 19:22:27.272362 28406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:37707 every 8 connection(s)
I20260810 19:22:27.273859 28192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:22:27.283097 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 28192
I20260810 19:22:27.283303 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:22:27.284080 7340 external_mini_cluster.cc:1428] Setting key 340ac5c6ef611974073871f9ef741dd6
Aug 10 19:22:27 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:27.310204 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:27.277537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:46127 (local address 127.7.43.1:49641)
0810 19:22:27.286323 (+ 8786us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:27.286388 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:27.288922 (+ 2534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:27.291446 (+ 2524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:27.291460 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:27.292223 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:27.293178 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:27.293213 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:27.295470 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:27.295476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:27.296751 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:27.296767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:27.298024 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:27.298033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:27.298308 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:27.299206 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:27.299233 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:27.301680 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:27.305443 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:27.305452 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:27.305470 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:27.306105 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:27.306589 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:27.306593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:27.306596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:27.306854 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:27.307513 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:27.307520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:27.307955 (+ 435us) client_negotiation.cc:769] Sending connection context
0810 19:22:27.308239 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 19:22:27.308764 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8427,"thread_start_us":11688,"threads_started":1}
I20260810 19:22:27.311338 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:27.276861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:49641 (local address 127.7.43.62:46127)
0810 19:22:27.290573 (+ 13712us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:27.290581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:27.290604 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:22:27.290674 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:27.290680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:27.290780 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:27.290912 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:27.293424 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:27.294713 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:27.296895 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:27.297463 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:27.297617 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:27.301944 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:27.301952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:27.301957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:27.302007 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:27.305277 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:27.306239 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:27.306244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:27.306247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:27.306441 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:27.306986 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:27.306990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:27.306993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:27.307302 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:27.307435 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:27.310780 (+ 3345us) server_negotiation.cc:299] Negotiation successful
0810 19:22:27.310994 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13550,"thread_start_us":13431,"threads_started":1}
I20260810 19:22:27.312944 28408 heartbeater.cc:347] Connected to a master server at 127.7.43.62:46127
I20260810 19:22:27.313251 28408 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:27.314121 28408 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
I20260810 19:22:27.316562 28057 ts_manager.cc:194] Registered new tserver with Master: 3d87ae5782294699b382c4c659c3fdb6 (127.7.43.1:37707)
I20260810 19:22:27.318521 28057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:49641
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:22:27.366194 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:46127
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:22:27 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389746, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
W20260810 19:22:27.852092 28420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:27.852527 28420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:27.852690 28420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:27.863996 28420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:27.864151 28420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:27.864363 28420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:22:27.880429 28420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.43.20:41763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:27.882278 28420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:27.884087 28420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:27.899641 28437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:27.899618 28438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:27.900378 28440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:27.900733 28420 server_base.cc:1034] running on GCE node
I20260810 19:22:27.901360 28420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:27.902434 28420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:27.903649 28420 hybrid_clock.cc:648] HybridClock initialized: now 1786389747903589 us; error 58 us; skew 500 ppm
I20260810 19:22:27.903998 28420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:27 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389747, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:27.907809 28420 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:22:27.909349 28420 webserver.cc:460] Webserver started at http://127.7.43.2:37885/ using document root <none> and password file <none>
I20260810 19:22:27.910075 28420 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:27.910182 28420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:27.910486 28420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:22:27 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389747, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:27.928184 28420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "75326a73b10244fba15f3676f06f8a0c"
format_stamp: "Formatted at 2026-08-10 19:22:27 on dist-test-slave-7lbf"
server_key: "a9c570ba5a008dd4e65c01f8f32893f3"
server_key_iv: "bfd7647036af18cfc6ce3198d16c32e4"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:27.929469 28420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "75326a73b10244fba15f3676f06f8a0c"
format_stamp: "Formatted at 2026-08-10 19:22:27 on dist-test-slave-7lbf"
server_key: "a9c570ba5a008dd4e65c01f8f32893f3"
server_key_iv: "bfd7647036af18cfc6ce3198d16c32e4"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:27.937474 28420 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260810 19:22:27.957603 28449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:27.959566 28420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:22:27.959770 28420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "75326a73b10244fba15f3676f06f8a0c"
format_stamp: "Formatted at 2026-08-10 19:22:27 on dist-test-slave-7lbf"
server_key: "a9c570ba5a008dd4e65c01f8f32893f3"
server_key_iv: "bfd7647036af18cfc6ce3198d16c32e4"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:27.960045 28420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:28.009850 28420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:28.015376 28420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:28.015749 28420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:28.017164 28420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:28.019140 28420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:28.019243 28420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:28.019428 28420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:28.019541 28420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:28.069255 28420 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45183
I20260810 19:22:28.069312 28637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45183 every 8 connection(s)
I20260810 19:22:28.071165 28420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:22:28.078161 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 28420
I20260810 19:22:28.078327 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:22:28.078933 7340 external_mini_cluster.cc:1428] Setting key 83ef5a90702aa7fecc762bd2d902b9d9
Aug 10 19:22:28 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389747, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:28.116377 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:28.074317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:44483 (local address 127.7.43.62:46127)
0810 19:22:28.089646 (+ 15329us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:28.089654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:28.095320 (+ 5666us) server_negotiation.cc:407] Connection header received
0810 19:22:28.097286 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:28.097292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:28.097404 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:28.097544 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:28.099227 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:28.099986 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:28.102523 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:28.103094 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:28.103212 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:28.109028 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:28.109035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:28.109040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:28.109078 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:28.112379 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:28.113388 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:28.113393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:28.113396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:28.113586 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:28.114132 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:28.114138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:28.114141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:28.114533 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:28.114660 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:28.115738 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0810 19:22:28.115988 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15154,"thread_start_us":15333,"threads_started":1}
I20260810 19:22:28.119099 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:28.074965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:46127 (local address 127.7.43.2:44483)
0810 19:22:28.095132 (+ 20167us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:28.095180 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:28.096989 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:28.097830 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:28.097841 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:28.098349 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:28.099056 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:28.099079 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:28.100140 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:28.100143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:28.102368 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:28.102382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:28.104740 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:28.104748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:28.105135 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:28.106004 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:28.106033 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:28.108742 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:28.112625 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:28.112638 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:28.112658 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:28.113253 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:28.113724 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:28.113728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:28.113731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:28.114006 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:28.114759 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:28.114767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:28.115266 (+ 499us) client_negotiation.cc:769] Sending connection context
0810 19:22:28.116966 (+ 1700us) client_negotiation.cc:240] Negotiation successful
0810 19:22:28.117620 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19865,"thread_start_us":19651,"threads_started":1}
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
I20260810 19:22:28.121110 28640 heartbeater.cc:347] Connected to a master server at 127.7.43.62:46127
I20260810 19:22:28.121423 28640 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:28.122296 28640 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
I20260810 19:22:28.123855 28057 ts_manager.cc:194] Registered new tserver with Master: 75326a73b10244fba15f3676f06f8a0c (127.7.43.2:45183)
I20260810 19:22:28.124678 28057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:44483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:22:28.157444 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:46127
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:28.322314 28408 heartbeater.cc:503] Master 127.7.43.62:46127 was elected leader, sending a full tablet report...
W20260810 19:22:28.813030 28651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:28.813414 28651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:28.813664 28651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:28.823865 28651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:28.824049 28651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:28.824273 28651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:22:28.837494 28651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.43.20:41763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:28.839095 28651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:28.840561 28651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:28.858618 28658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:28.858628 28662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:28.858500 28659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:28.859459 28651 server_base.cc:1034] running on GCE node
I20260810 19:22:28.860033 28651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:28.861104 28651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:28.862392 28651 hybrid_clock.cc:648] HybridClock initialized: now 1786389748862277 us; error 35 us; skew 500 ppm
I20260810 19:22:28.862812 28651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:28 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:28.870868 28651 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:22:28.872802 28651 webserver.cc:460] Webserver started at http://127.7.43.3:36513/ using document root <none> and password file <none>
I20260810 19:22:28.873696 28651 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:28.873878 28651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:28.874265 28651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:22:28 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:28.893455 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "94f7ff6417f4440c8bda171dcb4c7d9d"
format_stamp: "Formatted at 2026-08-10 19:22:28 on dist-test-slave-7lbf"
server_key: "29b1404f465bbd161028878ba5d82f30"
server_key_iv: "f8f4cbfa5f89510ee867d2205df50a69"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:28.895004 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "94f7ff6417f4440c8bda171dcb4c7d9d"
format_stamp: "Formatted at 2026-08-10 19:22:28 on dist-test-slave-7lbf"
server_key: "29b1404f465bbd161028878ba5d82f30"
server_key_iv: "f8f4cbfa5f89510ee867d2205df50a69"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:28.904397 28651 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.001s
I20260810 19:22:28.925333 28672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:28.927469 28651 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260810 19:22:28.927763 28651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "94f7ff6417f4440c8bda171dcb4c7d9d"
format_stamp: "Formatted at 2026-08-10 19:22:28 on dist-test-slave-7lbf"
server_key: "29b1404f465bbd161028878ba5d82f30"
server_key_iv: "f8f4cbfa5f89510ee867d2205df50a69"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:28.928151 28651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:28.945007 28651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:28.950574 28651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:28.951063 28651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:28.952574 28651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:28.954625 28651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:28.954744 28651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:28.954916 28651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:28.955159 28651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:29.029071 28651 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:39243
I20260810 19:22:29.029184 28856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:39243 every 8 connection(s)
I20260810 19:22:29.030773 28651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:22:29.040984 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 28651
I20260810 19:22:29.041172 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:22:29.041927 7340 external_mini_cluster.cc:1428] Setting key 039b6a656c71973c3a02ada18ff2051a
Aug 10 19:22:29 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389748, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:29.058557 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:29.034434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:59745 (local address 127.7.43.62:46127)
0810 19:22:29.035385 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:29.035392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:29.036867 (+ 1475us) server_negotiation.cc:407] Connection header received
0810 19:22:29.039475 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:29.039480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:29.039596 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:29.039692 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:29.041926 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.042953 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.045637 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.046392 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.046489 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:29.051158 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:29.051163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:29.051167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:29.051198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:29.054945 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.055694 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.055699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.055703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.055930 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.056582 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.056587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.056590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.057001 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:29.057187 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:29.057961 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 19:22:29.058186 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":554,"threads_started":1}
I20260810 19:22:29.059690 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:29.035234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:46127 (local address 127.7.43.3:59745)
0810 19:22:29.036683 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:29.036745 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:29.039233 (+ 2488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:29.039905 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:29.039913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:29.040533 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:29.041751 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:29.041781 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.043183 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.043189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:29.045453 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:29.045489 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.046646 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.046654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:29.047019 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:29.047794 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:29.047819 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:29.050883 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:29.055064 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:29.055071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:29.055083 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:29.055579 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:29.056047 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:29.056051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:29.056054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:29.056461 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:29.057125 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:29.057131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:29.057549 (+ 418us) client_negotiation.cc:769] Sending connection context
0810 19:22:29.057906 (+ 357us) client_negotiation.cc:240] Negotiation successful
0810 19:22:29.058366 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":817,"threads_started":1}
I20260810 19:22:29.061105 28857 heartbeater.cc:347] Connected to a master server at 127.7.43.62:46127
I20260810 19:22:29.061432 28857 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:29.062495 28857 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
I20260810 19:22:29.064154 28057 ts_manager.cc:194] Registered new tserver with Master: 94f7ff6417f4440c8bda171dcb4c7d9d (127.7.43.3:39243)
I20260810 19:22:29.065109 28057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:59745
I20260810 19:22:29.070678 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:22:29.073344 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 28192
Aug 10 19:22:29 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:22:29.097036 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:29.074508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.7.43.1:37707)
0810 19:22:29.076168 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:29.076175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:29.076198 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:22:29.076276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:29.076283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:29.076470 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:29.076655 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:29.079120 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.080590 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.082774 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.083651 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.083771 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:29.089498 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:29.089510 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:29.089562 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:29.089599 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:29.093787 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.094602 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.094611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.094614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.094851 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.095264 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.095268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.095270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.095676 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:29.096019 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:29.096247 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 19:22:29.096570 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510,"thread_start_us":1299,"threads_started":1}
I20260810 19:22:29.098564 28399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:29.132714 28640 heartbeater.cc:503] Master 127.7.43.62:46127 was elected leader, sending a full tablet report...
I20260810 19:22:29.182808 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 28192
I20260810 19:22:29.201314 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 28420
Aug 10 19:22:29 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:22:29.219748 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:29.202547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.7.43.2:45183)
0810 19:22:29.203618 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:29.203625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:29.203653 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:22:29.203720 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:29.203727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:29.203970 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:29.204106 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:29.206140 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.207221 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.208868 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.209412 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.209564 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:29.214150 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:29.214164 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:29.214179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:29.214236 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:29.217324 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.218017 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.218024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.218027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.218171 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.218466 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.218469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.218470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.218807 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:29.219054 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:29.219242 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 19:22:29.219468 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":581,"threads_started":1}
I20260810 19:22:29.221046 28624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:29.302532 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 28420
I20260810 19:22:29.328625 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 28651
Aug 10 19:22:29 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:22:29.344251 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:29.329617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.7.43.3:39243)
0810 19:22:29.330263 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:29.330271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:29.330298 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:22:29.330362 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:29.330368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:29.330600 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:29.330763 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:29.332143 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.333145 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.334481 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:29.335025 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:29.335130 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:29.339481 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:29.339492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:29.339506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:29.339541 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:29.342080 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.342660 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.342667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.342669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.342790 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:29.343068 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:29.343071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:29.343073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:29.343389 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:29.343628 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:29.343796 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 19:22:29.343981 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":347,"threads_started":1}
I20260810 19:22:29.345469 28849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:29.426362 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 28651
I20260810 19:22:29.447973 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 28005
I20260810 19:22:29.449190 28112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:29.554768 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 28005
I20260810 19:22:29.583155 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:46127
--webserver_interface=127.7.43.62
--webserver_port=40979
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:46127
--ranger_config_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:29.917160 28878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:29.917469 28878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:29.917635 28878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:29.925033 28878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:29.925156 28878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:29.925248 28878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:29.925328 28878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:29.925395 28878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:29.935930 28878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:46127
--ranger_config_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=40979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:29.937228 28878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:29.938575 28878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:29.949982 28887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:29.949940 28888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:29.950083 28892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:29.951156 28878 server_base.cc:1034] running on GCE node
I20260810 19:22:29.951952 28878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:29.953294 28878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:29.954499 28878 hybrid_clock.cc:648] HybridClock initialized: now 1786389749954442 us; error 44 us; skew 500 ppm
I20260810 19:22:29.954828 28878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:29 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:29.957855 28878 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:29.959204 28878 webserver.cc:460] Webserver started at http://127.7.43.62:40979/ using document root <none> and password file <none>
I20260810 19:22:29.959824 28878 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:29.959930 28878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:22:30 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389749, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:30.988319 28878 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:22:30.992735 28909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:30.994357 28878 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 19:22:30.994561 28878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "11c2ba66ddfc486fafb46f8300af70d4"
format_stamp: "Formatted at 2026-08-10 19:22:25 on dist-test-slave-7lbf"
server_key: "cbdf02f51e23817c4cc2c1b1ac8d70b1"
server_key_iv: "b0d55a5cbcb83e6b04c537384c9a1419"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:30.995673 28878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:31.012892 28878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:31.016758 28878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:31.017038 28878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:31.035717 28878 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:46127
I20260810 19:22:31.035738 28988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:46127 every 8 connection(s)
I20260810 19:22:31.037252 28878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:31.043767 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 28878
I20260810 19:22:31.044247 28989 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:22:31.044623 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:37707
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:46127
--webserver_port=40999
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:31.047708 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Bootstrap starting.
I20260810 19:22:31.066857 28989 log.cc:824] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:31.076397 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:22:31.076932 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Bootstrap complete.
I20260810 19:22:31.085858 28989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:31.086732 28989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c2ba66ddfc486fafb46f8300af70d4, State: Initialized, Role: FOLLOWER
I20260810 19:22:31.087383 28989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:31.087607 28989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:22:31.087726 28989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:22:31.087890 28989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:22:31.091801 28989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:31.092314 28989 leader_election.cc:302] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c2ba66ddfc486fafb46f8300af70d4; no voters:
I20260810 19:22:31.092926 28989 leader_election.cc:288] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:22:31.093082 28997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:22:31.094097 28997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [term 2 LEADER]: Becoming Leader. State: Replica: 11c2ba66ddfc486fafb46f8300af70d4, State: Running, Role: LEADER
I20260810 19:22:31.094732 28997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } }
I20260810 19:22:31.095176 28989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:31.097158 28999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c2ba66ddfc486fafb46f8300af70d4. Latest consensus state: current_term: 2 leader_uuid: "11c2ba66ddfc486fafb46f8300af70d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } } }
I20260810 19:22:31.097745 28998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "11c2ba66ddfc486fafb46f8300af70d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c2ba66ddfc486fafb46f8300af70d4" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 46127 } } }
I20260810 19:22:31.098026 28998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:31.098471 28999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:31.103281 29003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:31.110033 29003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:31.111737 29003 catalog_manager.cc:1259] Loaded cluster ID: 05ea8796048d471c8fc19546eb8adf09
I20260810 19:22:31.111954 29003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:22:31.116573 29003 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:22:31.118445 29003 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Loaded TSK: 0
I20260810 19:22:31.120201 29003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:22:31.122952 28989 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 19:22:31.502532 28991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:31.503094 28991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:31.503312 28991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:31.513738 28991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:31.514002 28991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:31.514247 28991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:22:31.526732 28991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:37707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=40999
--tserver_master_addrs=127.7.43.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:31.528160 28991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:31.529474 28991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:31.557541 29059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:31.557495 29051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:31.558395 29054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:31.558856 28991 server_base.cc:1034] running on GCE node
I20260810 19:22:31.559499 28991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:31.560523 28991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:31.561764 28991 hybrid_clock.cc:648] HybridClock initialized: now 1786389751561683 us; error 49 us; skew 500 ppm
I20260810 19:22:31.562157 28991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:31 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389751, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:31.566118 28991 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:22:31.573136 28991 webserver.cc:460] Webserver started at http://127.7.43.1:40999/ using document root <none> and password file <none>
I20260810 19:22:31.574005 28991 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:31.574152 28991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:22:31 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389751, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:31.608644 28991 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.004s
I20260810 19:22:31.612514 29067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:31.614128 28991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:22:31.614334 28991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "3d87ae5782294699b382c4c659c3fdb6"
format_stamp: "Formatted at 2026-08-10 19:22:26 on dist-test-slave-7lbf"
server_key: "1e20efecc54b335e2d125bd3c55e37fc"
server_key_iv: "1e99b5d5cbf30d93f8ad666a3314f6cb"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:31.615456 28991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:31.646471 28991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:31.650667 28991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:31.650961 28991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:31.652170 28991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:31.653788 28991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:31.653887 28991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:31.653995 28991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:31.654074 28991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:31.696636 28991 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:37707
I20260810 19:22:31.696740 29254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:37707 every 8 connection(s)
I20260810 19:22:31.698415 28991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:22:31.700632 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 28991
I20260810 19:22:31.701447 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:45183
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:46127
--webserver_port=37885
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:22:31 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389751, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:31.752172 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:31.705302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:46127 (local address 127.7.43.1:50893)
0810 19:22:31.718615 (+ 13313us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:31.718688 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:31.721336 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:31.731842 (+ 10506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:31.731856 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:31.732502 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:31.733376 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:31.733407 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:31.735773 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:31.735779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:31.737840 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:31.737869 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:31.739232 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:31.739240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:31.739530 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:31.740245 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:31.740276 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:31.743013 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:31.747188 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:31.747197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:31.747213 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:31.747787 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:31.748363 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:31.748368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:31.748372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:31.748628 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:31.749595 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:31.749603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:31.749994 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 19:22:31.750258 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 19:22:31.750810 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12924,"thread_start_us":12670,"threads_started":1}
I20260810 19:22:31.753849 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:31.706476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:50893 (local address 127.7.43.62:46127)
0810 19:22:31.729864 (+ 23388us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:31.729891 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:31.729957 (+ 66us) server_negotiation.cc:407] Connection header received
0810 19:22:31.730483 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:31.730520 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:31.731125 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:31.731609 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:31.733665 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:31.735558 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:31.738001 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:31.738977 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:31.739146 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:31.743323 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:31.743334 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:31.743354 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:31.743409 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:31.747027 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:31.747993 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:31.748002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:31.748010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:31.748224 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:31.748777 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:31.748782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:31.748785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:31.749416 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:31.751407 (+ 1991us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:31.751726 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 19:22:31.752160 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23019,"thread_start_us":23344,"threads_started":1}
I20260810 19:22:31.757115 29255 heartbeater.cc:347] Connected to a master server at 127.7.43.62:46127
I20260810 19:22:31.757452 29255 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:31.758397 29255 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
W20260810 19:22:31.761950 29255 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:46127 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:22:31.762192 29255 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:31.762477 29255 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
I20260810 19:22:31.763862 29255 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:31.764150 29255 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
W20260810 19:22:31.765388 29255 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:22:32.110666 29266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:32.110966 29266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:32.111143 29266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:32.118659 29266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:32.118798 29266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:32.118979 29266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:22:32.133976 29266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:45183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=37885
--tserver_master_addrs=127.7.43.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:32.135489 29266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:32.136837 29266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:32.150748 29287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:32.150849 29291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:32.150930 29289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:32.152542 29266 server_base.cc:1034] running on GCE node
I20260810 19:22:32.153389 29266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:32.154706 29266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:32.155962 29266 hybrid_clock.cc:648] HybridClock initialized: now 1786389752155893 us; error 55 us; skew 500 ppm
I20260810 19:22:32.156374 29266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:32.160555 29266 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:22:32.164676 29266 webserver.cc:460] Webserver started at http://127.7.43.2:37885/ using document root <none> and password file <none>
I20260810 19:22:32.165721 29266 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:32.165896 29266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:32.192435 29266 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:22:32.195362 29307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:32.196905 29266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:22:32.197073 29266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "75326a73b10244fba15f3676f06f8a0c"
format_stamp: "Formatted at 2026-08-10 19:22:27 on dist-test-slave-7lbf"
server_key: "a9c570ba5a008dd4e65c01f8f32893f3"
server_key_iv: "bfd7647036af18cfc6ce3198d16c32e4"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:32.198099 29266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:32.220113 29266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:32.225428 29266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:32.225798 29266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:32.226946 29266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:32.228461 29266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:32.228559 29266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:32.228672 29266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:32.228744 29266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:32.336730 29266 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45183
I20260810 19:22:32.336789 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45183 every 8 connection(s)
I20260810 19:22:32.338357 29266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:22:32.343695 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 29266
I20260810 19:22:32.344632 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.43.20:41763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:39243
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:46127
--webserver_port=36513
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:32.396294 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:32.342816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:52803 (local address 127.7.43.62:46127)
0810 19:22:32.359070 (+ 16254us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:32.359077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:32.360345 (+ 1268us) server_negotiation.cc:407] Connection header received
0810 19:22:32.362775 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:32.362782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:32.362886 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:32.363007 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:32.365240 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:32.366474 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:32.370846 (+ 4372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:32.372641 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:32.372764 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:32.388389 (+ 15625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:32.388398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:32.388403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:32.388456 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:32.392127 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:32.393222 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:32.393229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:32.393232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:32.393427 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:32.394064 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:32.394070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:32.394074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:32.394466 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:32.394600 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:32.395643 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0810 19:22:32.395881 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16099,"thread_start_us":15975,"threads_started":1}
I20260810 19:22:32.398881 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:32.342095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:46127 (local address 127.7.43.2:52803)
0810 19:22:32.359706 (+ 17611us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:32.359772 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:32.362233 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:32.363542 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:32.363555 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:32.364089 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:32.365000 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:32.365032 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:32.366709 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:32.366714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:32.370628 (+ 3914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:32.370660 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:32.373180 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:32.373187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:32.373506 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:32.382835 (+ 9329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:32.382882 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:32.387989 (+ 5107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:32.392348 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:32.392359 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:32.392382 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:32.393058 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:32.393629 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:32.393634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:32.393637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:32.393912 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:32.394694 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:32.394704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:32.395155 (+ 451us) client_negotiation.cc:769] Sending connection context
0810 19:22:32.396751 (+ 1596us) client_negotiation.cc:240] Negotiation successful
0810 19:22:32.397410 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17282,"thread_start_us":17027,"threads_started":1}
I20260810 19:22:32.400729 29506 heartbeater.cc:347] Connected to a master server at 127.7.43.62:46127
I20260810 19:22:32.401077 29506 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:32.402060 29506 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
I20260810 19:22:32.404191 28937 ts_manager.cc:194] Registered new tserver with Master: 75326a73b10244fba15f3676f06f8a0c (127.7.43.2:45183)
I20260810 19:22:32.406464 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:52803
I20260810 19:22:32.765563 29255 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:32.766012 29255 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
I20260810 19:22:32.767306 28937 ts_manager.cc:194] Registered new tserver with Master: 3d87ae5782294699b382c4c659c3fdb6 (127.7.43.1:37707)
I20260810 19:22:32.768460 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:50893
W20260810 19:22:32.784267 29513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:32.784566 29513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:32.784700 29513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:32.795459 29513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:32.795598 29513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:32.795784 29513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:22:32.806565 29513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.43.20:41763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:39243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=36513
--tserver_master_addrs=127.7.43.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:32.808072 29513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:32.809784 29513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:32.825413 29522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:32.825815 29521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:32.827338 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:32.828145 29513 server_base.cc:1034] running on GCE node
I20260810 19:22:32.828959 29513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:32.831796 29513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:32.833120 29513 hybrid_clock.cc:648] HybridClock initialized: now 1786389752833049 us; error 53 us; skew 500 ppm
I20260810 19:22:32.833563 29513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:32.837770 29513 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:22:32.839860 29513 webserver.cc:460] Webserver started at http://127.7.43.3:36513/ using document root <none> and password file <none>
I20260810 19:22:32.840889 29513 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:32.841110 29513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
Aug 10 19:22:32 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:32.876994 29513 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.008s
I20260810 19:22:32.880860 29539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:32.882845 29513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:22:32.883054 29513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "94f7ff6417f4440c8bda171dcb4c7d9d"
format_stamp: "Formatted at 2026-08-10 19:22:28 on dist-test-slave-7lbf"
server_key: "29b1404f465bbd161028878ba5d82f30"
server_key_iv: "f8f4cbfa5f89510ee867d2205df50a69"
server_key_version: "kuduclusterkey@0"
I20260810 19:22:32.884459 29513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:32.931929 29513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:32.936345 29513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:32.936653 29513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:32.937824 29513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:32.939421 29513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:32.939529 29513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:32.939656 29513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:32.939738 29513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:32.985240 29513 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:39243
I20260810 19:22:32.985297 29721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:39243 every 8 connection(s)
I20260810 19:22:32.986732 29513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:22:32.991456 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 29513
Aug 10 19:22:33 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:33.030983 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:32.997005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.7.43.62:46127)
0810 19:22:33.012112 (+ 15107us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.012119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.012143 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:22:33.012217 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.012223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.012318 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.014642 (+ 2324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:33.014818 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.015955 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.018035 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.018671 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.018788 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.022490 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:33.022497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:33.022514 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:33.022553 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:33.026473 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.027402 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.027407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.027411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.027582 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.028021 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.028026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.028030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.028380 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:33.028642 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.030244 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.030561 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14917,"thread_start_us":8642,"threads_started":1}
I20260810 19:22:33.039499 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:32.989640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:57437 (local address 127.7.43.62:46127)
0810 19:22:33.005914 (+ 16274us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.005920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.012815 (+ 6895us) server_negotiation.cc:407] Connection header received
0810 19:22:33.015148 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.015154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.015265 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.015403 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:33.018932 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.020133 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.023269 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.024044 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.024164 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.029056 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:33.029064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:33.029070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:33.029120 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:33.033002 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.034062 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.034067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.034071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.034208 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.034773 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.034778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.034781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.037684 (+ 2903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:33.037850 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.038850 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.039100 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16138,"thread_start_us":22128,"threads_started":1}
I20260810 19:22:33.042479 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:32.990344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:46127 (local address 127.7.43.3:57437)
0810 19:22:33.012599 (+ 22255us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.012659 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:33.014836 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:33.015981 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:33.015996 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:33.016645 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:33.018654 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.018685 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.020456 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.020462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.022696 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.023103 (+ 407us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.024614 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.024623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.024957 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.025710 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:33.025746 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:33.028720 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:33.033186 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:33.033196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:33.033215 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:33.033914 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:33.034368 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:33.034373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:33.034376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:33.034631 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:33.037953 (+ 3322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:33.037962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:33.038406 (+ 444us) client_negotiation.cc:769] Sending connection context
0810 19:22:33.039715 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0810 19:22:33.040475 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21906,"thread_start_us":21633,"threads_started":1}
I20260810 19:22:33.044314 29722 heartbeater.cc:347] Connected to a master server at 127.7.43.62:46127
I20260810 19:22:33.044660 29722 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:33.045627 29722 heartbeater.cc:511] Master 127.7.43.62:46127 requested a full tablet report, sending...
I20260810 19:22:33.047267 28937 ts_manager.cc:194] Registered new tserver with Master: 94f7ff6417f4440c8bda171dcb4c7d9d (127.7.43.3:39243)
I20260810 19:22:33.048341 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:57437
I20260810 19:22:33.059383 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:22:33.086023 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.068275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.7.43.62:46127)
0810 19:22:33.068470 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.068476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.068721 (+ 245us) server_negotiation.cc:407] Connection header received
0810 19:22:33.068918 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.068924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.069011 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.069128 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:33.071229 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.072378 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.074415 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.075083 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.075219 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.078463 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:33.078470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:33.078474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:33.078516 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:33.082028 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.082943 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.082949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.082953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.083123 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.083578 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.083583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.083587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.083947 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:33.084188 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.085391 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.085670 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:22:33.092646 28937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:22:33.104651 28937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:22:33.155489 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.124399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:39243 (local address 127.0.0.1:36580)
0810 19:22:33.137351 (+ 12952us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.137413 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:33.137679 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:33.144328 (+ 6649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:33.144335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:33.144365 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:33.144750 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.144766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.149313 (+ 4547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.149320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.152067 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.152084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.154390 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.154397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.154617 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.154681 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 19:22:33.154831 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 19:22:33.155032 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12623,"mutex_wait_us":21,"thread_start_us":22874,"threads_started":1}
I20260810 19:22:33.159711 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.125053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.7.43.3:39243)
0810 19:22:33.143293 (+ 18240us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.143303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.143340 (+ 37us) server_negotiation.cc:407] Connection header received
0810 19:22:33.143451 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.143459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.143735 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.143978 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:33.145000 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.149171 (+ 4171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.152283 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.153834 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.153951 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.154199 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.159192 (+ 4993us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.159372 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16832,"thread_start_us":19993,"threads_started":1}
I20260810 19:22:33.160199 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.125138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.7.43.2:45183)
0810 19:22:33.138373 (+ 13235us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.138383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.147136 (+ 8753us) server_negotiation.cc:407] Connection header received
0810 19:22:33.147461 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.147470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.147703 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.147901 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:33.151500 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.153172 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.157012 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.158553 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.158658 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.158864 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.159552 (+ 688us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.159799 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13055,"thread_start_us":12913,"threads_started":1}
I20260810 19:22:33.162606 29751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.124905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45183 (local address 127.0.0.1:39850)
0810 19:22:33.147008 (+ 22103us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.147033 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:33.147248 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:33.148167 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:33.148172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:33.148196 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:33.149675 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.150866 (+ 1191us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.153386 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.153391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.156828 (+ 3437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.156846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.159080 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.159086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.159333 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.159372 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 19:22:33.160446 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0810 19:22:33.162184 (+ 1738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21986,"mutex_wait_us":1,"thread_start_us":13495,"threads_started":1}
I20260810 19:22:33.167156 29396 tablet_service.cc:1467] Processing CreateTablet for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 (DEFAULT_TABLE table=test-table [id=7312d8a8a1144495a670497dc82ac551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:33.168306 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.148119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.7.43.1:37707)
0810 19:22:33.148851 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.148861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.155972 (+ 7111us) server_negotiation.cc:407] Connection header received
0810 19:22:33.156072 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.156080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.156316 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.156514 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:33.157324 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.158890 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.162165 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.166477 (+ 4312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.166609 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.166834 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.167690 (+ 856us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.167935 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":324,"threads_started":1}
I20260810 19:22:33.168815 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:33.169186 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.149410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36210)
0810 19:22:33.155703 (+ 6293us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.155723 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:33.155843 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:33.156839 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:33.156843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:33.156910 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:33.157077 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.157089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.159066 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.159071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.161955 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.161987 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.167243 (+ 5256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.167250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.167459 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.167509 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 19:22:33.168705 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0810 19:22:33.168858 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6192,"mutex_wait_us":830,"threads_started":1}
I20260810 19:22:33.175634 29148 tablet_service.cc:1467] Processing CreateTablet for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 (DEFAULT_TABLE table=test-table [id=7312d8a8a1144495a670497dc82ac551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:33.177150 29148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:22:33 dist-test-slave-7lbf krb5kdc[27294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389752, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for HTTP/127.7.43.20@KRBTEST.COM
I20260810 19:22:33.180526 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 (DEFAULT_TABLE table=test-table [id=7312d8a8a1144495a670497dc82ac551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:33.182085 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:33.200841 29764 tablet_bootstrap.cc:492] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Bootstrap starting.
I20260810 19:22:33.205678 29766 tablet_bootstrap.cc:492] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Bootstrap starting.
I20260810 19:22:33.206532 29764 tablet_bootstrap.cc:654] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:33.207576 29764 log.cc:824] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:33.209221 29766 tablet_bootstrap.cc:654] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:33.210487 29766 log.cc:824] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:33.211545 29764 tablet_bootstrap.cc:492] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: No bootstrap required, opened a new log
I20260810 19:22:33.211848 29764 ts_tablet_manager.cc:1397] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 19:22:33.219319 29771 tablet_bootstrap.cc:492] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: Bootstrap starting.
I20260810 19:22:33.221856 29764 raft_consensus.cc:348] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.222491 29764 raft_consensus.cc:374] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:33.222604 29764 raft_consensus.cc:729] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75326a73b10244fba15f3676f06f8a0c, State: Initialized, Role: FOLLOWER
I20260810 19:22:33.223280 29771 tablet_bootstrap.cc:654] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:33.223282 29764 consensus_queue.cc:260] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.224620 29771 log.cc:824] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:33.225240 29766 tablet_bootstrap.cc:492] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: No bootstrap required, opened a new log
I20260810 19:22:33.225605 29766 ts_tablet_manager.cc:1397] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260810 19:22:33.236972 29766 raft_consensus.cc:348] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.237501 29766 raft_consensus.cc:374] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:33.239483 29766 raft_consensus.cc:729] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f7ff6417f4440c8bda171dcb4c7d9d, State: Initialized, Role: FOLLOWER
I20260810 19:22:33.240309 29766 consensus_queue.cc:260] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.241845 29722 heartbeater.cc:503] Master 127.7.43.62:46127 was elected leader, sending a full tablet report...
I20260810 19:22:33.243134 29766 ts_tablet_manager.cc:1428] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Time spent starting tablet: real 0.017s user 0.006s sys 0.008s
I20260810 19:22:33.245617 29771 tablet_bootstrap.cc:492] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: No bootstrap required, opened a new log
I20260810 19:22:33.246114 29771 ts_tablet_manager.cc:1397] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.018s
I20260810 19:22:33.254455 29506 heartbeater.cc:503] Master 127.7.43.62:46127 was elected leader, sending a full tablet report...
I20260810 19:22:33.254581 29771 raft_consensus.cc:348] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.255057 29771 raft_consensus.cc:374] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:33.255159 29771 raft_consensus.cc:729] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d87ae5782294699b382c4c659c3fdb6, State: Initialized, Role: FOLLOWER
I20260810 19:22:33.255268 29764 ts_tablet_manager.cc:1428] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Time spent starting tablet: real 0.043s user 0.021s sys 0.023s
I20260810 19:22:33.256107 29771 consensus_queue.cc:260] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.257663 29255 heartbeater.cc:503] Master 127.7.43.62:46127 was elected leader, sending a full tablet report...
I20260810 19:22:33.259114 29771 ts_tablet_manager.cc:1428] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260810 19:22:33.290501 29778 raft_consensus.cc:482] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:22:33.290903 29778 raft_consensus.cc:504] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.293219 29778 leader_election.cc:288] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d87ae5782294699b382c4c659c3fdb6 (127.7.43.1:37707), 75326a73b10244fba15f3676f06f8a0c (127.7.43.2:45183)
I20260810 19:22:33.302505 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.294655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:57165 (local address 127.7.43.1:37707)
0810 19:22:33.294848 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.294853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.294872 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:22:33.294941 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.294947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.295018 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.295101 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:33.296440 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.297568 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.299958 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.301134 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.301271 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.301391 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.302127 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.302254 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:22:33.303045 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.293829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.7.43.3:57165)
0810 19:22:33.294279 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.294300 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:33.294432 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:33.296032 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:33.296038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:33.296066 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:33.296254 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.296267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.297734 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.297740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.299708 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.299725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.301475 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.301481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.301696 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.301730 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:22:33.301828 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:22:33.301970 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260810 19:22:33.304201 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cbdc6b6f33450ab55ae6c0da6ed7e4" candidate_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d87ae5782294699b382c4c659c3fdb6" is_pre_election: true
I20260810 19:22:33.304842 29175 raft_consensus.cc:2454] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f7ff6417f4440c8bda171dcb4c7d9d in term 0.
I20260810 19:22:33.305660 29542 leader_election.cc:302] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d87ae5782294699b382c4c659c3fdb6, 94f7ff6417f4440c8bda171dcb4c7d9d; no voters:
I20260810 19:22:33.306110 29778 raft_consensus.cc:2781] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:22:33.306258 29778 raft_consensus.cc:482] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:22:33.306361 29778 raft_consensus.cc:3037] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:33.308051 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.295902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:50291 (local address 127.7.43.2:45183)
0810 19:22:33.296082 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.296088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.300163 (+ 4075us) server_negotiation.cc:407] Connection header received
0810 19:22:33.301412 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.301418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.301534 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.301609 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:33.302358 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.303469 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.305744 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.306903 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.307012 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.307104 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.307666 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.307800 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:22:33.308741 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.295495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45183 (local address 127.7.43.3:50291)
0810 19:22:33.300065 (+ 4570us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.300083 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:33.300276 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:33.301808 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:33.301812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:33.301848 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:33.302182 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.302194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.303652 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.303657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.305567 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:33.305581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.307286 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.307292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:33.307476 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.307509 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:22:33.308283 (+ 774us) client_negotiation.cc:240] Negotiation successful
0810 19:22:33.308417 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4449,"thread_start_us":7157,"threads_started":1}
I20260810 19:22:33.309711 29778 raft_consensus.cc:504] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.309751 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cbdc6b6f33450ab55ae6c0da6ed7e4" candidate_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75326a73b10244fba15f3676f06f8a0c" is_pre_election: true
I20260810 19:22:33.310344 29426 raft_consensus.cc:2454] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f7ff6417f4440c8bda171dcb4c7d9d in term 0.
I20260810 19:22:33.310811 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cbdc6b6f33450ab55ae6c0da6ed7e4" candidate_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d87ae5782294699b382c4c659c3fdb6"
I20260810 19:22:33.311033 29175 raft_consensus.cc:3037] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:33.314327 29175 raft_consensus.cc:2454] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f7ff6417f4440c8bda171dcb4c7d9d in term 1.
I20260810 19:22:33.314757 29542 leader_election.cc:302] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d87ae5782294699b382c4c659c3fdb6, 94f7ff6417f4440c8bda171dcb4c7d9d; no voters:
I20260810 19:22:33.317446 29778 leader_election.cc:288] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [CANDIDATE]: Term 1 election: Requested vote from peers 3d87ae5782294699b382c4c659c3fdb6 (127.7.43.1:37707), 75326a73b10244fba15f3676f06f8a0c (127.7.43.2:45183)
I20260810 19:22:33.317660 29778 raft_consensus.cc:2781] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:33.317796 29778 raft_consensus.cc:686] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 1 LEADER]: Becoming Leader. State: Replica: 94f7ff6417f4440c8bda171dcb4c7d9d, State: Running, Role: LEADER
I20260810 19:22:33.318037 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cbdc6b6f33450ab55ae6c0da6ed7e4" candidate_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75326a73b10244fba15f3676f06f8a0c"
I20260810 19:22:33.318280 29426 raft_consensus.cc:3037] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:33.318317 29778 consensus_queue.cc:237] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } }
I20260810 19:22:33.321785 29426 raft_consensus.cc:2454] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f7ff6417f4440c8bda171dcb4c7d9d in term 1.
I20260810 19:22:33.335543 28936 catalog_manager.cc:5495] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f7ff6417f4440c8bda171dcb4c7d9d (127.7.43.3). New cstate: current_term: 1 leader_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f7ff6417f4440c8bda171dcb4c7d9d" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 39243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 } health_report { overall_health: UNKNOWN } } }
W20260810 19:22:33.347445 29507 tablet.cc:2369] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:22:33.396327 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.388881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.7.43.3:39243)
0810 19:22:33.389071 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.389077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.389278 (+ 201us) server_negotiation.cc:407] Connection header received
0810 19:22:33.389553 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.389559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.389654 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.389797 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:22:33.390229 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.391211 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.393283 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.393964 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.394118 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.394830 (+ 712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:22:33.395572 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:22:33.395725 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.395919 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.396065 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:22:33.418615 29426 raft_consensus.cc:1261] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 1 FOLLOWER]: Refusing update from remote peer 94f7ff6417f4440c8bda171dcb4c7d9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:33.418800 29175 raft_consensus.cc:1261] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 1 FOLLOWER]: Refusing update from remote peer 94f7ff6417f4440c8bda171dcb4c7d9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:33.419699 29784 consensus_queue.cc:1035] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75326a73b10244fba15f3676f06f8a0c" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:33.420570 29778 consensus_queue.cc:1035] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d87ae5782294699b382c4c659c3fdb6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:33.442689 29787 mvcc.cc:205] Tried to move back new op lower bound from 7317052429990531072 to 7317052429596450816. Current Snapshot: MvccSnapshot[applied={T|T < 7317052429990531072}]
I20260810 19:22:33.447184 29788 mvcc.cc:205] Tried to move back new op lower bound from 7317052429990531072 to 7317052429596450816. Current Snapshot: MvccSnapshot[applied={T|T < 7317052429990531072}]
W20260810 19:22:33.473752 29256 tablet.cc:2369] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:22:33.477842 29790 mvcc.cc:205] Tried to move back new op lower bound from 7317052429990531072 to 7317052429596450816. Current Snapshot: MvccSnapshot[applied={T|T < 7317052429990531072}]
I20260810 19:22:33.500191 28936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:22:33.500666 28936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:22:33.506690 28936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11c2ba66ddfc486fafb46f8300af70d4: Sending DeleteTablet for 3 replicas of tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4
I20260810 19:22:33.508565 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:33 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:36580
I20260810 19:22:33.508829 29148 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:33 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:36210
I20260810 19:22:33.510496 29396 tablet_service.cc:1514] Processing DeleteTablet for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:33 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:39850
I20260810 19:22:33.520848 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 28991
I20260810 19:22:33.525954 29813 tablet_replica.cc:330] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: stopping tablet replica
I20260810 19:22:33.526584 29813 raft_consensus.cc:2229] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:22:33.527643 29813 raft_consensus.cc:2258] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:33.530215 29813 ts_tablet_manager.cc:1905] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:33.539986 29813 ts_tablet_manager.cc:1918] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:33.540222 29813 log.cc:1196] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/14cbdc6b6f33450ab55ae6c0da6ed7e4
I20260810 19:22:33.540961 29813 ts_tablet_manager.cc:1939] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 94f7ff6417f4440c8bda171dcb4c7d9d: Deleting consensus metadata
I20260810 19:22:33.543184 28913 catalog_manager.cc:4841] TS 94f7ff6417f4440c8bda171dcb4c7d9d (127.7.43.3:39243): tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 (table test-table [id=7312d8a8a1144495a670497dc82ac551]) successfully deleted
I20260810 19:22:33.556320 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.523833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.7.43.1:37707)
0810 19:22:33.524029 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.524036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.524057 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:22:33.524124 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.524130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.524232 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.524352 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:33.527254 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.528393 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.531679 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.532372 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.532506 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.545493 (+ 12987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:33.545566 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:33.545589 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:33.545643 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:33.551271 (+ 5628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.553059 (+ 1788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.553070 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.553073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.553345 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.553943 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.553949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.553953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.554591 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:33.555068 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.555391 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.555633 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:22:33.557364 29250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:33.560649 29815 tablet_replica.cc:330] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: stopping tablet replica
I20260810 19:22:33.561317 29815 raft_consensus.cc:2229] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:22:33.562451 29815 raft_consensus.cc:2258] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:33.565153 29815 ts_tablet_manager.cc:1905] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:33.574520 29815 ts_tablet_manager.cc:1918] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:33.574777 29815 log.cc:1196] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/14cbdc6b6f33450ab55ae6c0da6ed7e4
I20260810 19:22:33.575860 29815 ts_tablet_manager.cc:1939] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 75326a73b10244fba15f3676f06f8a0c: Deleting consensus metadata
I20260810 19:22:33.578562 28914 catalog_manager.cc:4841] TS 75326a73b10244fba15f3676f06f8a0c (127.7.43.2:45183): tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4 (table test-table [id=7312d8a8a1144495a670497dc82ac551]) successfully deleted
I20260810 19:22:33.719630 29814 tablet_replica.cc:330] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6: stopping tablet replica
I20260810 19:22:33.720058 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 28991
I20260810 19:22:33.720124 29814 raft_consensus.cc:2229] T 14cbdc6b6f33450ab55ae6c0da6ed7e4 P 3d87ae5782294699b382c4c659c3fdb6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 19:22:33.748834 28913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:37707 (error 108)
I20260810 19:22:33.749963 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 29266
I20260810 19:22:33.772156 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.751721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.7.43.2:45183)
0810 19:22:33.751925 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.751933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.751955 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:22:33.752027 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.752033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.752137 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.752261 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:33.754524 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.755562 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.757727 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.758283 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.758424 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.762217 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:33.762241 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:33.762265 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:33.762320 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:33.767636 (+ 5316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.768704 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.768716 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.768720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.768894 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.769378 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.769384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.769400 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.769921 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:33.770230 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.771525 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.771763 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 19:22:33.773197 29494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:33.773721 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.766787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36228)
0810 19:22:33.773302 (+ 6515us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.773465 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6362,"threads_started":1}
W20260810 19:22:33.774430 28913 catalog_manager.cc:4584] TS 3d87ae5782294699b382c4c659c3fdb6 (127.7.43.1:37707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14cbdc6b6f33450ab55ae6c0da6ed7e4: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
I20260810 19:22:33.828334 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.827750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36242)
0810 19:22:33.828043 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.828153 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260810 19:22:33.829612 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.829107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36256)
0810 19:22:33.829333 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.829414 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 19:22:33.890585 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.889811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36258)
0810 19:22:33.890249 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.890378 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289}
I20260810 19:22:33.892050 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.891538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36270)
0810 19:22:33.891778 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.891873 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 19:22:33.922915 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 29266
I20260810 19:22:33.946666 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 29513
I20260810 19:22:33.965162 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.947578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.7.43.3:39243)
0810 19:22:33.948333 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:33.948337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:33.948353 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:22:33.948411 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:33.948415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:33.948486 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:33.948583 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:33.949994 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.950863 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.952189 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:33.952649 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:33.952769 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:33.958538 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:33.958555 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:33.958573 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:33.958611 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:33.961874 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.962907 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.962916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.962919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.963087 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:33.963555 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:33.963559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:33.963561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:33.963968 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:33.964280 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:33.964581 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 19:22:33.964806 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":493,"threads_started":1}
I20260810 19:22:33.966022 29716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:33.972836 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.972316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36282)
0810 19:22:33.972552 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.972663 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 19:22:33.974154 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:33.973646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36290)
0810 19:22:33.973868 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:33.973968 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 19:22:34.089453 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 29513
I20260810 19:22:34.110924 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 28878
I20260810 19:22:34.112136 28981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:34.222555 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:34.222021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:37707 (local address 127.0.0.1:36304)
0810 19:22:34.222236 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:34.222351 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 19:22:34.222599 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 28878
I20260810 19:22:34.249873 7340 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 19:22:34.584246 7340 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 19:22:34.584448 7340 mini_ranger.cc:66] Stopping Ranger...
I20260810 19:22:34.928653 7340 mini_ranger.cc:68] Stopped Ranger
2026-08-10 19:22:34.928 UTC [27342] LOG: received smart shutdown request
2026-08-10 19:22:34.932 UTC [27342] LOG: background worker "logical replication launcher" (PID 27354) exited with exit code 1
2026-08-10 19:22:34.933 UTC [27346] LOG: shutting down
2026-08-10 19:22:35.083 UTC [27342] LOG: database system is shut down
2026-08-10T19:22:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54792 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:22:35 dist-test-slave-7lbf krb5kdc[29840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:22:35 dist-test-slave-7lbf krb5kdc[29840](info): set up 2 sockets
Aug 10 19:22:35 dist-test-slave-7lbf krb5kdc[29840](info): commencing operation
krb5kdc: starting...
W20260810 19:22:36.449857 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.002s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:22:36 dist-test-slave-7lbf krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:22:36.715243 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:41663
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:41663,127.7.43.61:37629,127.7.43.60:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:37.055984 29862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:37.056267 29862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:37.056366 29862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:37.063431 29862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:37.063530 29862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:37.063583 29862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:37.063640 29862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:37.063684 29862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:37.073144 29862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:41663,127.7.43.61:37629,127.7.43.60:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:37.074376 29862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:37.075608 29862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:37.086810 29871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:37.086851 29869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:37.086886 29868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:37.087507 29862 server_base.cc:1034] running on GCE node
I20260810 19:22:37.088229 29862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:37.089566 29862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:37.090756 29862 hybrid_clock.cc:648] HybridClock initialized: now 1786389757090683 us; error 68 us; skew 500 ppm
I20260810 19:22:37.091087 29862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:37 dist-test-slave-7lbf krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:37.094115 29862 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:37.095513 29862 webserver.cc:460] Webserver started at http://127.7.43.62:42037/ using document root <none> and password file <none>
I20260810 19:22:37.096190 29862 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:37.096304 29862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:37.096596 29862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:37.099646 29862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e"
format_stamp: "Formatted at 2026-08-10 19:22:37 on dist-test-slave-7lbf"
server_key: "c52545c8e40b70b4ba62d38d6b07f3a4"
server_key_iv: "7dadda7cc15fb0c6c49fcbc088a07e32"
server_key_version: "encryptionkey@0"
I20260810 19:22:37.100441 29862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e"
format_stamp: "Formatted at 2026-08-10 19:22:37 on dist-test-slave-7lbf"
server_key: "c52545c8e40b70b4ba62d38d6b07f3a4"
server_key_iv: "7dadda7cc15fb0c6c49fcbc088a07e32"
server_key_version: "encryptionkey@0"
I20260810 19:22:37.105425 29862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260810 19:22:37.108774 29884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:37.110129 29862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:22:37.110280 29862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e"
format_stamp: "Formatted at 2026-08-10 19:22:37 on dist-test-slave-7lbf"
server_key: "c52545c8e40b70b4ba62d38d6b07f3a4"
server_key_iv: "7dadda7cc15fb0c6c49fcbc088a07e32"
server_key_version: "encryptionkey@0"
I20260810 19:22:37.110460 29862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:37.130240 29862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:37.134306 29862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:37.134588 29862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:37.151818 29862 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:41663
I20260810 19:22:37.151850 29984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:41663 every 8 connection(s)
I20260810 19:22:37.153179 29862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:37.156814 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:37.159955 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 29862
I20260810 19:22:37.160203 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:22:37.161007 7340 external_mini_cluster.cc:1428] Setting key ef0f6fe2ce215a9e9048f9a7412dd98e
I20260810 19:22:37.164893 29985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:22:37 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:37.189157 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.168522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:41663 (local address 127.0.0.1:51830)
0810 19:22:37.169348 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.169395 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:37.171633 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:37.172575 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:37.172581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:37.173311 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:37.173828 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.173851 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.175963 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.175967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.177089 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.177101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.178309 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.178315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.178446 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.179133 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:37.179155 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:37.181422 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:37.185167 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.185177 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.185188 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.185629 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.186112 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.186116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.186119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.186284 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.187271 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:37.187283 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:37.187486 (+ 203us) client_negotiation.cc:769] Sending connection context
0810 19:22:37.187686 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 19:22:37.188095 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":363,"threads_started":1}
I20260810 19:22:37.189183 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.169843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.7.43.62:41663)
0810 19:22:37.170510 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:37.170518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:37.170595 (+ 77us) server_negotiation.cc:407] Connection header received
0810 19:22:37.172034 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:37.172042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:37.172330 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:37.172536 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:37.174041 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.175805 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.177233 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.178103 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.178294 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.181698 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:37.181710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:37.181727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:37.181762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:37.185024 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.185754 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.185763 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.185766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.185984 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.186395 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.186400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.186402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.187142 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:37.187374 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:37.187901 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 19:22:37.188267 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":463,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.7.43.61@KRBTEST.COM; defaulting to no policy
I20260810 19:22:37.192395 29985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.61@KRBTEST.COM" created.
I20260810 19:22:37.193454 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.193059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41356)
0810 19:22:37.193216 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.193334 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 19:22:37.193765 29895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
I20260810 19:22:37.195310 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.194910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41366)
0810 19:22:37.195074 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.195153 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:22:37.195499 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:37629: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab.
Entry for principal kudu/127.7.43.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab.
I20260810 19:22:37.222996 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:37629
--webserver_interface=127.7.43.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab
--principal=kudu/127.7.43.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:41663,127.7.43.61:37629,127.7.43.60:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:37.244932 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } attempt: 1
I20260810 19:22:37.246169 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.245735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41376)
0810 19:22:37.245921 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.246022 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 19:22:37.247175 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.246823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41378)
0810 19:22:37.246966 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.247046 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:22:37.247366 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:37629: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
I20260810 19:22:37.315575 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } attempt: 2
I20260810 19:22:37.316632 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.316177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41382)
0810 19:22:37.316400 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.316489 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 19:22:37.317651 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.317265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41398)
0810 19:22:37.317404 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.317479 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:22:37.317811 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:37629: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
I20260810 19:22:37.409057 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } attempt: 3
I20260810 19:22:37.410281 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.409848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41406)
0810 19:22:37.410042 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.410144 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 19:22:37.411309 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.410940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41418)
0810 19:22:37.411097 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.411165 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 19:22:37.411520 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:37629: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
I20260810 19:22:37.554750 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } attempt: 4
I20260810 19:22:37.555840 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.555343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41422)
0810 19:22:37.555627 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.555711 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260810 19:22:37.556686 29997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260810 19:22:37.556869 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.556529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:41438)
0810 19:22:37.556668 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.556752 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:22:37.556970 29997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:37.557052 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:37629: Network error: Client connection negotiation failed: client connection to 127.7.43.61:37629: connect: Connection refused (error 111)
W20260810 19:22:37.557127 29997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:37.564208 29997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:37.564319 29997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:37.564405 29997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:37.564483 29997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:37.564546 29997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:37.574232 29997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:41663,127.7.43.61:37629,127.7.43.60:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab
--principal=kudu/127.7.43.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.43.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:37.575555 29997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:37.576870 29997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:37.588016 30006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:37.587926 30013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:37.587973 30009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:37.589115 29997 server_base.cc:1034] running on GCE node
I20260810 19:22:37.589975 29997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:37.591352 29997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:37.592549 29997 hybrid_clock.cc:648] HybridClock initialized: now 1786389757592498 us; error 44 us; skew 500 ppm
I20260810 19:22:37.592883 29997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:37 dist-test-slave-7lbf krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:37.595896 29997 init.cc:377] Logged in from keytab as kudu/127.7.43.61@KRBTEST.COM (short username kudu)
I20260810 19:22:37.597256 29997 webserver.cc:460] Webserver started at http://127.7.43.61:35823/ using document root <none> and password file <none>
I20260810 19:22:37.597966 29997 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:37.598074 29997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:37.598363 29997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:37.601260 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data/instance:
uuid: "3ac0960c54fc43af9b6215b5678cedb9"
format_stamp: "Formatted at 2026-08-10 19:22:37 on dist-test-slave-7lbf"
server_key: "66ab0d1d2da208642549432ba390500e"
server_key_iv: "ecc144a9cb54e164658b40959752b95a"
server_key_version: "encryptionkey@0"
I20260810 19:22:37.602105 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal/instance:
uuid: "3ac0960c54fc43af9b6215b5678cedb9"
format_stamp: "Formatted at 2026-08-10 19:22:37 on dist-test-slave-7lbf"
server_key: "66ab0d1d2da208642549432ba390500e"
server_key_iv: "ecc144a9cb54e164658b40959752b95a"
server_key_version: "encryptionkey@0"
I20260810 19:22:37.607334 29997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:22:37.610816 30024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:37.612144 29997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:22:37.612300 29997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal
uuid: "3ac0960c54fc43af9b6215b5678cedb9"
format_stamp: "Formatted at 2026-08-10 19:22:37 on dist-test-slave-7lbf"
server_key: "66ab0d1d2da208642549432ba390500e"
server_key_iv: "ecc144a9cb54e164658b40959752b95a"
server_key_version: "encryptionkey@0"
I20260810 19:22:37.612475 29997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:37.632465 29997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:37.636169 29997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:37.636416 29997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:37.655818 29997 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.61:37629
I20260810 19:22:37.655850 30119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.61:37629 every 8 connection(s)
I20260810 19:22:37.657183 29997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
I20260810 19:22:37.657660 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 29997
I20260810 19:22:37.657846 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-1/wal/instance
I20260810 19:22:37.658404 7340 external_mini_cluster.cc:1428] Setting key 4c8127370788224e0f63690189ba7a24
I20260810 19:22:37.661139 30120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:37.667702 30120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:22:37 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.43.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.60@KRBTEST.COM" created.
I20260810 19:22:37.687933 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.670326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.7.43.62:41663)
0810 19:22:37.670554 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:37.670560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:37.671808 (+ 1248us) server_negotiation.cc:407] Connection header received
0810 19:22:37.673686 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:37.673692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:37.673775 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:37.673882 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:37.675332 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.676237 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.677751 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.678283 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.678401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.681954 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:37.681961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:37.681966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:37.682007 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:37.685053 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.685776 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.685779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.685782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.685895 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.686251 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.686254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.686256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.686576 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:37.686731 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:37.687433 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 19:22:37.687638 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:22:37.688704 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.670791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:41663 (local address 127.0.0.1:58670)
0810 19:22:37.671618 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.671669 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:37.673392 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:37.674089 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:37.674097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:37.674601 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:37.675161 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.675178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.676375 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.676379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.677621 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.677633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.678461 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.678466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.678654 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.679270 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:37.679292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:37.681723 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:37.685192 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.685199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.685212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.685682 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.685969 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.685973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.685976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.686164 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.686700 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:37.686708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:37.687059 (+ 351us) client_negotiation.cc:769] Sending connection context
0810 19:22:37.687277 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 19:22:37.687643 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":361,"threads_started":1}
I20260810 19:22:37.690395 30120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:22:37 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:22:37.704734 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.691123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:56018)
0810 19:22:37.691273 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.691292 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:37.691385 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:37.692614 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:37.692618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:37.692978 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:37.693117 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.693125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.694849 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.694852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.695665 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.695675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.696629 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.696633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.696800 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.697312 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:37.697328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:37.699339 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:37.702281 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.702286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.702290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.702728 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.703050 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.703056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.703059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.703270 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.703829 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:37.703833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:37.703976 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 19:22:37.704123 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 19:22:37.704395 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 19:22:37.704787 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.691405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.7.43.61:37629)
0810 19:22:37.692127 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:37.692135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:37.692165 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:22:37.692217 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:37.692224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:37.692458 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:37.692604 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:37.693259 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.694706 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.695790 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.696504 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.696623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.699530 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:37.699540 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:37.699553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:37.699583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:37.702159 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.702835 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.702841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.702843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.702953 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.703386 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.703389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.703391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.703709 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:37.703863 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:37.704237 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 19:22:37.704497 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":537,"thread_start_us":371,"threads_started":1}
I20260810 19:22:37.707329 30120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:22:37.708338 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.707928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44938)
0810 19:22:37.708078 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.708194 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 19:22:37.708613 30028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:37.710109 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.709809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44942)
0810 19:22:37.709929 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.710013 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:22:37.710326 30120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab.
Entry for principal kudu/127.7.43.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab.
I20260810 19:22:37.721064 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:40527
--webserver_interface=127.7.43.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab
--principal=kudu/127.7.43.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:41663,127.7.43.61:37629,127.7.43.60:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:37.759617 30120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 1
I20260810 19:22:37.760852 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.760368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44950)
0810 19:22:37.760581 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.760696 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:22:37.761790 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.761438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44952)
0810 19:22:37.761610 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.761677 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:22:37.761979 30120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:37.830193 30120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 2
I20260810 19:22:37.831363 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.830908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44954)
0810 19:22:37.831106 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.831208 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 19:22:37.832479 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.832099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44960)
0810 19:22:37.832252 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.832341 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:22:37.832710 30120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:37.856285 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } attempt: 5
Aug 10 19:22:37 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:22:37.870563 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.857047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:56032)
0810 19:22:37.857264 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.857283 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:37.857391 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:37.857759 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:37.857763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:37.858162 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:37.858342 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.858350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.859678 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.859682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.860522 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:37.860532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.861505 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.861509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:37.861678 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.862344 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:37.862363 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:37.864612 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:37.868298 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.868301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.868304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.868675 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.869050 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:37.869053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:37.869055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:37.869182 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:37.869830 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:37.869834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:37.869976 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 19:22:37.870090 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 19:22:37.870288 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 19:22:37.870793 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.857168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.7.43.61:37629)
0810 19:22:37.857385 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:37.857392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:37.857415 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:22:37.857511 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:37.857548 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:37.857632 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:37.857737 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:37.858500 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.859556 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.860669 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:37.861354 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:37.861498 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:37.864813 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:37.864820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:37.864825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:37.864866 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:37.868144 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.868789 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.868795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.868798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.868954 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:37.869274 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:37.869280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:37.869283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:37.869727 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:37.869880 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:37.870233 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 19:22:37.870474 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:22:37.871836 29985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:22:37.872812 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.872454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44964)
0810 19:22:37.872603 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.872675 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:22:37.873693 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.873349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44980)
0810 19:22:37.873482 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.873590 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:22:37.873881 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:37.923954 30120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 3
I20260810 19:22:37.925112 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.924584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44990)
0810 19:22:37.924818 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.924923 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:22:37.925107 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 1
I20260810 19:22:37.926255 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.925754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:44992)
0810 19:22:37.925967 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.926074 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 19:22:37.926808 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.926315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45006)
0810 19:22:37.926569 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.926665 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260810 19:22:37.927125 30120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:37.927306 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.926949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45012)
0810 19:22:37.927106 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.927185 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 19:22:37.927512 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:37.995738 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 2
I20260810 19:22:37.996959 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.996365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45024)
0810 19:22:37.996607 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.996744 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 19:22:37.998039 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:37.997622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45036)
0810 19:22:37.997792 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:37.997884 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:22:37.998277 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
W20260810 19:22:38.060643 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:38.060947 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:38.061046 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:38.068167 30136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:38.068281 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:38.068329 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:38.068390 30136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:38.068434 30136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:38.070351 30120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 4
I20260810 19:22:38.071656 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.071087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45048)
0810 19:22:38.071354 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.071470 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 19:22:38.072659 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.072297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45064)
0810 19:22:38.072453 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.072535 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 19:22:38.072877 30120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:38.078922 30136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:41663,127.7.43.61:37629,127.7.43.60:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab
--principal=kudu/127.7.43.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.43.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:38.080215 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:38.081450 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:38.092860 30145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:38.092936 30147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:38.092902 30149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:38.094071 30136 server_base.cc:1034] running on GCE node
I20260810 19:22:38.094841 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:38.096235 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:38.097436 30136 hybrid_clock.cc:648] HybridClock initialized: now 1786389758097379 us; error 50 us; skew 500 ppm
I20260810 19:22:38.097796 30136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389758, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:38.100814 30136 init.cc:377] Logged in from keytab as kudu/127.7.43.60@KRBTEST.COM (short username kudu)
I20260810 19:22:38.102334 30136 webserver.cc:460] Webserver started at http://127.7.43.60:33015/ using document root <none> and password file <none>
I20260810 19:22:38.102993 30136 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:38.103092 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:38.103350 30136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:38.104470 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 3
I20260810 19:22:38.105568 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.104991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45076)
0810 19:22:38.105241 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.105341 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:22:38.106575 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.106162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45088)
0810 19:22:38.106340 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.106425 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 19:22:38.106827 29985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:40527: Network error: Client connection negotiation failed: client connection to 127.7.43.60:40527: connect: Connection refused (error 111)
I20260810 19:22:38.106731 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data/instance:
uuid: "9231db14801840dbbd197f3119c48487"
format_stamp: "Formatted at 2026-08-10 19:22:38 on dist-test-slave-7lbf"
server_key: "1fab2f16b8cc726f59d238667a2df1fb"
server_key_iv: "cb49cc5ba30c7b234aa426ad7bf9e4a3"
server_key_version: "encryptionkey@0"
I20260810 19:22:38.107534 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal/instance:
uuid: "9231db14801840dbbd197f3119c48487"
format_stamp: "Formatted at 2026-08-10 19:22:38 on dist-test-slave-7lbf"
server_key: "1fab2f16b8cc726f59d238667a2df1fb"
server_key_iv: "cb49cc5ba30c7b234aa426ad7bf9e4a3"
server_key_version: "encryptionkey@0"
I20260810 19:22:38.112517 30136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:22:38.116139 30166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:38.117590 30136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:22:38.117736 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal
uuid: "9231db14801840dbbd197f3119c48487"
format_stamp: "Formatted at 2026-08-10 19:22:38 on dist-test-slave-7lbf"
server_key: "1fab2f16b8cc726f59d238667a2df1fb"
server_key_iv: "cb49cc5ba30c7b234aa426ad7bf9e4a3"
server_key_version: "encryptionkey@0"
I20260810 19:22:38.117899 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:38.135533 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:38.139652 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:38.139896 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:38.158341 30136 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.60:40527
I20260810 19:22:38.158360 30246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.60:40527 every 8 connection(s)
I20260810 19:22:38.159732 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
I20260810 19:22:38.163511 30248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:38.167359 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 30136
I20260810 19:22:38.167582 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786389541863099-7340-0/minicluster-data/master-2/wal/instance
I20260810 19:22:38.168210 7340 external_mini_cluster.cc:1428] Setting key 3581053c92e6584573f8124c5007dbd1
I20260810 19:22:38.171380 30248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389758, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:38.189149 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.175111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.7.43.62:41663)
0810 19:22:38.175331 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.175337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.176124 (+ 787us) server_negotiation.cc:407] Connection header received
0810 19:22:38.176294 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.176300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.176380 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.176483 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.177597 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.178282 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.179675 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.180115 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.180221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.183730 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.183735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.183738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.183771 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.186207 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.186852 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.186857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.186859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.186993 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.187361 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.187365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.187367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.187622 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.187774 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.188347 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.188579 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:22:38.189985 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.175136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:41663 (local address 127.0.0.1:58672)
0810 19:22:38.175970 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.176012 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:38.176177 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:38.176548 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:38.176553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:38.177144 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:38.177400 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.177410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.178445 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.178449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.179534 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.179547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.180327 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.180334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.180486 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.181043 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:38.181064 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:38.183544 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:38.186326 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.186338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.186351 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.186739 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.187098 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.187103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.187107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.187258 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.187755 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:38.187763 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:38.187977 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 19:22:38.188227 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 19:22:38.188740 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":2,"thread_start_us":437,"threads_started":1}
I20260810 19:22:38.190004 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.170210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.7.43.60:40527)
0810 19:22:38.171007 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.171020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.171062 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:22:38.172592 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.172615 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.173049 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.173420 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.175075 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.176831 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.178282 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.179026 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.179194 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.183392 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.183402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.183413 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.183445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.186206 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.186894 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.186900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.186902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.187052 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.187428 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.187432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.187434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.187938 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.188224 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.188558 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.188873 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":584,"thread_start_us":364,"threads_started":1}
I20260810 19:22:38.191537 30248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389758, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:38.204408 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.192459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.7.43.61:37629)
0810 19:22:38.192615 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.192621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.192643 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:22:38.192705 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.192710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.192788 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.192890 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.193724 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.194403 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.195596 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.196017 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.196106 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.199255 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.199260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.199263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.199294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.201848 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.202576 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.202580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.202582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.202705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.203080 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.203084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.203086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.203378 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.203504 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.203977 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.204141 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:22:38.204523 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.192314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:37629 (local address 127.0.0.1:56036)
0810 19:22:38.192488 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.192507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:38.192598 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:38.192939 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:38.192945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:38.193368 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:38.193577 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.193590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.194551 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.194556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.195453 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.195467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.196149 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.196155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.196379 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.196944 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:38.196960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:38.199082 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:38.201976 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.201982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.201986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.202471 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.202810 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.202813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.202816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.202968 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.203500 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:38.203506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:38.203741 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 19:22:38.203899 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 19:22:38.204170 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:22:38.205552 30248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:22:38.207060 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.192868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.7.43.62:41663)
0810 19:22:38.193040 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.193045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.193060 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:22:38.193100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.193103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.193152 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.193220 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.194549 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.195262 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.196772 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.197224 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.197314 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.201326 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.201331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.201334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.201364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.204321 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.205090 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.205094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.205096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.205221 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.205636 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.205641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.205644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.205991 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.206241 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.206562 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.206771 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389758, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:22:38.218951 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.206303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45114)
0810 19:22:38.206483 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.206501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:38.206586 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:38.206914 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:38.206918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:38.207756 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:38.207931 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.207943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.208969 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.208974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.209962 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.209975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.210760 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.210766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.210960 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.211481 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:38.211499 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:38.213552 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:38.216458 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.216462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.216464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.216952 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.217291 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.217296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.217299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.217453 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.218051 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:38.218055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:38.218168 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 19:22:38.218288 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 19:22:38.218564 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:22:38.218999 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.206506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.7.43.60:40527)
0810 19:22:38.206648 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.206654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.206671 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:22:38.206715 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.206720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.206784 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.206875 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.208069 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.208849 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.210098 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.210626 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.210714 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.213727 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.213733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.213738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.213773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.216342 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.217059 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.217063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.217066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.217183 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.217597 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.217601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.217604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.217940 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.218091 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.218407 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.218720 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":53}
I20260810 19:22:38.223178 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.208698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.7.43.61:37629)
0810 19:22:38.208831 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.208834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.208849 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:22:38.208952 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.208956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.209023 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.209116 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.210365 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.211284 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.212527 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.212969 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.213086 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.217637 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.217643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.217647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.217682 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.220505 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.221166 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.221172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.221175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.221344 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.221811 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.221816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.221819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.222178 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.222471 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.222636 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.222859 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:22:38.225162 30248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: Bootstrap starting.
I20260810 19:22:38.227984 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:22:38.228660 30248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:38.229763 30248 log.cc:824] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:38.232594 30248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: No bootstrap required, opened a new log
I20260810 19:22:38.243831 30248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.244513 30248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:38.244689 30248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231db14801840dbbd197f3119c48487, State: Initialized, Role: FOLLOWER
I20260810 19:22:38.245653 30248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.252985 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.233142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.7.43.61:37629)
0810 19:22:38.233329 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.233335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.234225 (+ 890us) server_negotiation.cc:407] Connection header received
0810 19:22:38.234519 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.234525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.234606 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.234676 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.236931 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.238129 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.240365 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.241046 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.241148 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.245481 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.245488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.245493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.245555 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.249234 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.250244 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.250250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.250254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.250452 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.251021 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.251027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.251031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.251464 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.251728 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.252374 (+ 646us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.252638 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:22:38.252848 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.253252 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.232973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.7.43.62:41663)
0810 19:22:38.233161 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.233166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.234457 (+ 1291us) server_negotiation.cc:407] Connection header received
0810 19:22:38.234721 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.234727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.234814 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.234895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.237282 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.238491 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.240803 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.241560 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.241660 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.246334 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.246342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.246346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.246391 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.249796 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.250753 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.250759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.250762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.250937 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.251454 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.251460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.251463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.251832 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.252071 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.252646 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.252886 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:22:38.255074 30248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:38.255206 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.260762 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.234727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.7.43.60:40527)
0810 19:22:38.234917 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.234923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.243770 (+ 8847us) server_negotiation.cc:407] Connection header received
0810 19:22:38.243921 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.243926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.244007 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.244104 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.246225 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.247378 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.249357 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.250111 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.250207 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.253702 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.253708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.253712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.253753 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.257439 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.258435 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.258439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.258442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.258594 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.259083 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.259088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.259091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.259523 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.259803 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.260133 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.260400 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 19:22:38.269920 30287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:22:38.270054 30287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:22:38.262645 30136 master_runner.cc:186] Error getting master registration for 127.7.43.62:41663: OK, instance_id { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" instance_seqno: 1786389757146943 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:22:38.271688 30136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:22:38.284029 29985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 4
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
I20260810 19:22:38.298689 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.284612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.7.43.60:40527)
0810 19:22:38.284781 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.284787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.284809 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:22:38.284947 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.284953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.285036 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.285116 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.285967 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.286748 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.288580 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.289028 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.289149 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.293018 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.293023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.293027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.293058 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.295829 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.296693 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.296697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.296699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.296836 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.297276 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.297279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.297281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.297595 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.297707 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.298132 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.298307 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:22:38.298785 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.284532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45136)
0810 19:22:38.284703 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.284724 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:38.284828 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:38.285162 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:38.285167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:38.285635 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:38.285829 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.285839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.286900 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.286906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.288407 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.288424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.289175 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.289181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.289383 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.290130 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:38.290155 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:38.292837 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:38.295990 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.295996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.296000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.296584 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.296957 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.296962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.296966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.297156 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.297704 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:38.297709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:38.297893 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 19:22:38.298051 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 19:22:38.298379 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:22:38.305302 29985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: Bootstrap starting.
I20260810 19:22:38.309659 29985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:38.311131 29985 log.cc:824] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:38.314618 29985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: No bootstrap required, opened a new log
I20260810 19:22:38.325645 29985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.326090 29985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:38.326223 29985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf4ca4a1f3a4eff9658cc78c9f9475e, State: Initialized, Role: FOLLOWER
I20260810 19:22:38.326920 29985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.327867 30292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.328212 30292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.329365 29985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:38.335259 30292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:22:38.335556 30292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.337137 30292 leader_election.cc:288] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac0960c54fc43af9b6215b5678cedb9 (127.7.43.61:37629), 9231db14801840dbbd197f3119c48487 (127.7.43.60:40527)
I20260810 19:22:38.338821 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231db14801840dbbd197f3119c48487" is_pre_election: true
I20260810 19:22:38.339536 30209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf4ca4a1f3a4eff9658cc78c9f9475e in term 0.
I20260810 19:22:38.341637 29895 leader_election.cc:302] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf4ca4a1f3a4eff9658cc78c9f9475e, 9231db14801840dbbd197f3119c48487; no voters:
I20260810 19:22:38.342206 30292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:22:38.342361 30292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:22:38.342468 30292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 0 FOLLOWER]: Advancing to term 1
W20260810 19:22:38.345073 30304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:22:38.345403 30304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:22:38.348841 30292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.351511 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231db14801840dbbd197f3119c48487"
I20260810 19:22:38.351756 30209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:38.355244 30209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf4ca4a1f3a4eff9658cc78c9f9475e in term 1.
I20260810 19:22:38.355938 30292 leader_election.cc:288] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [CANDIDATE]: Term 1 election: Requested vote from peers 3ac0960c54fc43af9b6215b5678cedb9 (127.7.43.61:37629), 9231db14801840dbbd197f3119c48487 (127.7.43.60:40527)
I20260810 19:22:38.356442 29895 leader_election.cc:302] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf4ca4a1f3a4eff9658cc78c9f9475e, 9231db14801840dbbd197f3119c48487; no voters:
I20260810 19:22:38.356786 30292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:38.344084 29862 master_runner.cc:186] Error getting master registration for 127.7.43.61:37629: OK, instance_id { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" instance_seqno: 1786389757650768 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:22:38.361799 29862 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:22:38.362406 30292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [term 1 LEADER]: Becoming Leader. State: Replica: 3cf4ca4a1f3a4eff9658cc78c9f9475e, State: Running, Role: LEADER
I20260810 19:22:38.363065 30292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.366532 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf4ca4a1f3a4eff9658cc78c9f9475e. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.366992 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:38.368017 30313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:38.370462 30313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:38.372109 30120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } attempt: 5
Aug 10 19:22:38 dist-test-slave-7lbf krb5kdc[29840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389757, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
I20260810 19:22:38.387203 30209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [term 1 FOLLOWER]: Refusing update from remote peer 3cf4ca4a1f3a4eff9658cc78c9f9475e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 19:22:38.387959 29903 tablet.cc:2369] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:22:38.389467 30308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:22:38.393879 30175 tablet.cc:2369] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:22:38.403824 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.372774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:40527 (local address 127.0.0.1:45138)
0810 19:22:38.373161 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:38.373182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:38.373304 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:38.374072 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:38.374077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:38.374617 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:38.374826 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.374838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.376471 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.376477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.377932 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:38.377950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.379229 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.379241 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:38.379461 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.382594 (+ 3133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:38.382647 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:38.387234 (+ 4587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:38.399285 (+ 12051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.399292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.399296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.399859 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.401758 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:38.401764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:38.401768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:38.401964 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:38.402727 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:38.402733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:38.402915 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 19:22:38.403068 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 19:22:38.403402 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260810 19:22:38.405055 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:38.373554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.7.43.60:40527)
0810 19:22:38.373740 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:38.373746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:38.373765 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:22:38.373827 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:38.373832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:38.373911 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:38.374016 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:38.375057 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.376232 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.378150 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:38.378841 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:38.378976 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:38.387663 (+ 8687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:38.387672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:38.387678 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:38.387722 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:38.399152 (+ 11430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.401373 (+ 2221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.401378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.401381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.401603 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:38.402138 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:38.402143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:38.402147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:38.402578 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:38.404248 (+ 1670us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:38.404390 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 19:22:38.404656 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:22:38.411235 30120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: Bootstrap starting.
I20260810 19:22:38.413406 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf4ca4a1f3a4eff9658cc78c9f9475e. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.414824 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.416733 30120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:38.419006 30120 log.cc:824] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:38.424824 30318 mvcc.cc:205] Tried to move back new op lower bound from 7317052450342531072 to 7317052450260217856. Current Snapshot: MvccSnapshot[applied={T|T < 7317052450342531072}]
I20260810 19:22:38.434039 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.434365 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:38.439234 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.439488 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.441007 30120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: No bootstrap required, opened a new log
I20260810 19:22:38.450177 30313 catalog_manager.cc:1347] Generated new cluster ID: d42be3ed3426458e872d561fc2ba4b51
I20260810 19:22:38.450304 30313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:22:38.452028 30120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.452450 30120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:38.452553 30120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac0960c54fc43af9b6215b5678cedb9, State: Initialized, Role: FOLLOWER
I20260810 19:22:38.453208 30120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } }
I20260810 19:22:38.455777 30120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:38.455806 30070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac0960c54fc43af9b6215b5678cedb9" is_pre_election: true
I20260810 19:22:38.456496 30070 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf4ca4a1f3a4eff9658cc78c9f9475e in term 0.
I20260810 19:22:38.457981 30069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac0960c54fc43af9b6215b5678cedb9"
I20260810 19:22:38.458268 30069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:38.466331 30069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf4ca4a1f3a4eff9658cc78c9f9475e in term 1.
I20260810 19:22:38.468677 30068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [term 1 FOLLOWER]: Refusing update from remote peer 3cf4ca4a1f3a4eff9658cc78c9f9475e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:38.471626 30308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:38.471868 30313 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:22:38.473867 30313 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:22:38.454581 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.478993 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.488780 30313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e: Generated new TSK 0
I20260810 19:22:38.490020 30313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:22:38.502385 30331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf4ca4a1f3a4eff9658cc78c9f9475e. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.502887 30331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.502813 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.503093 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:22:38.506867 30292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cf4ca4a1f3a4eff9658cc78c9f9475e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 41663 } } peers { permanent_uuid: "3ac0960c54fc43af9b6215b5678cedb9" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 37629 } } peers { permanent_uuid: "9231db14801840dbbd197f3119c48487" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 40527 } } }
I20260810 19:22:38.507182 30292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf4ca4a1f3a4eff9658cc78c9f9475e [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:38.528689 30358 catalog_manager.cc:1259] Loaded cluster ID: d42be3ed3426458e872d561fc2ba4b51
I20260810 19:22:38.528872 30358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: loading cluster ID for follower catalog manager: success
I20260810 19:22:38.538173 30358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: acquiring CA information for follower catalog manager: success
I20260810 19:22:38.540736 30358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ac0960c54fc43af9b6215b5678cedb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:22:39.272202 30287 catalog_manager.cc:1259] Loaded cluster ID: d42be3ed3426458e872d561fc2ba4b51
I20260810 19:22:39.272316 30287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: loading cluster ID for follower catalog manager: success
I20260810 19:22:39.274798 30287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: acquiring CA information for follower catalog manager: success
I20260810 19:22:39.276516 30287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9231db14801840dbbd197f3119c48487: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:22:39.649187 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 29862
I20260810 19:22:39.650280 29977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:39.758250 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 29862
I20260810 19:22:39.785614 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 29997
I20260810 19:22:39.786743 30110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:39.891557 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 29997
I20260810 19:22:39.916510 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 30136
I20260810 19:22:39.917691 30242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:40.023479 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 30136
2026-08-10T19:22:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4827 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:22:40 dist-test-slave-7lbf krb5kdc[30373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:22:40 dist-test-slave-7lbf krb5kdc[30373](info): set up 2 sockets
Aug 10 19:22:40 dist-test-slave-7lbf krb5kdc[30373](info): commencing operation
krb5kdc: starting...
W20260810 19:22:41.269402 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.008s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:22:41 dist-test-slave-7lbf krb5kdc[30373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:22:41.546794 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40433
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:40433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:41.879892 30398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:41.880192 30398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:41.880348 30398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:41.887544 30398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:41.887658 30398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:41.887734 30398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:41.887810 30398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:41.887871 30398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:41.897686 30398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:40433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:40433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:41.898983 30398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:41.900272 30398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:41.911533 30404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:41.911593 30405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:41.912034 30409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:41.912386 30398 server_base.cc:1034] running on GCE node
I20260810 19:22:41.913136 30398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:41.914587 30398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:41.915802 30398 hybrid_clock.cc:648] HybridClock initialized: now 1786389761915747 us; error 33 us; skew 500 ppm
I20260810 19:22:41.916128 30398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:41 dist-test-slave-7lbf krb5kdc[30373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389761, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:41.919075 30398 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:41.920499 30398 webserver.cc:460] Webserver started at http://127.7.43.62:39315/ using document root <none> and password file <none>
I20260810 19:22:41.921181 30398 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:41.921295 30398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:41.921648 30398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:41.924556 30398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "2381c45f6121472c9aab4f167d4dadce"
format_stamp: "Formatted at 2026-08-10 19:22:41 on dist-test-slave-7lbf"
server_key: "22a58d7a219adbc650e2722a52ea829b"
server_key_iv: "4d4761d22c6ac44e0dc1e6d45953c56f"
server_key_version: "encryptionkey@0"
I20260810 19:22:41.925276 30398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "2381c45f6121472c9aab4f167d4dadce"
format_stamp: "Formatted at 2026-08-10 19:22:41 on dist-test-slave-7lbf"
server_key: "22a58d7a219adbc650e2722a52ea829b"
server_key_iv: "4d4761d22c6ac44e0dc1e6d45953c56f"
server_key_version: "encryptionkey@0"
I20260810 19:22:41.930245 30398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:22:41.933665 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:41.934922 30398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:22:41.935087 30398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "2381c45f6121472c9aab4f167d4dadce"
format_stamp: "Formatted at 2026-08-10 19:22:41 on dist-test-slave-7lbf"
server_key: "22a58d7a219adbc650e2722a52ea829b"
server_key_iv: "4d4761d22c6ac44e0dc1e6d45953c56f"
server_key_version: "encryptionkey@0"
I20260810 19:22:41.935261 30398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:41.960632 30398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:41.964638 30398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:41.964922 30398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:41.982478 30398 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:40433
I20260810 19:22:41.982506 30492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:40433 every 8 connection(s)
I20260810 19:22:41.983913 30398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:41.988075 30494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:41.992285 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 30398
I20260810 19:22:41.992516 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:22:41.993369 7340 external_mini_cluster.cc:1428] Setting key 088fa7500bb0f1ec7ac8580078c0a8b1
I20260810 19:22:41.999739 30494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce: Bootstrap starting.
I20260810 19:22:42.002837 30494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:42.003882 30494 log.cc:824] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:42.006937 30494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce: No bootstrap required, opened a new log
Aug 10 19:22:42 dist-test-slave-7lbf krb5kdc[30373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:42.015056 30494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2381c45f6121472c9aab4f167d4dadce" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40433 } }
I20260810 19:22:42.015441 30494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:42.015568 30494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2381c45f6121472c9aab4f167d4dadce, State: Initialized, Role: FOLLOWER
I20260810 19:22:42.016222 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:41.995338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.7.43.62:40433)
0810 19:22:41.996100 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:41.996112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:41.996152 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:22:41.998490 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:41.998513 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:41.998878 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:41.999232 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:42.000767 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:42.002170 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:42.003501 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:42.004223 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:42.004385 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:42.008723 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:42.008731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:42.008744 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:42.008783 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:42.011973 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:42.012699 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:42.012706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:42.012713 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:42.012889 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:42.013286 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:42.013290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:42.013295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:42.013918 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:42.014214 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:42.014660 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 19:22:42.015086 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":343,"threads_started":1}
I20260810 19:22:42.016191 30494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2381c45f6121472c9aab4f167d4dadce" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40433 } }
I20260810 19:22:42.016449 30494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:22:42.016584 30494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:22:42.016752 30494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:42.019244 30494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2381c45f6121472c9aab4f167d4dadce" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40433 } }
I20260810 19:22:42.019786 30494 leader_election.cc:302] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2381c45f6121472c9aab4f167d4dadce; no voters:
I20260810 19:22:42.019974 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:22:42.020395 30494 leader_election.cc:288] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:22:42.020517 30504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:42.021095 30504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [term 1 LEADER]: Becoming Leader. State: Replica: 2381c45f6121472c9aab4f167d4dadce, State: Running, Role: LEADER
I20260810 19:22:42.021695 30504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2381c45f6121472c9aab4f167d4dadce" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40433 } }
I20260810 19:22:42.023533 30494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:42.024526 30505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2381c45f6121472c9aab4f167d4dadce. Latest consensus state: current_term: 1 leader_uuid: "2381c45f6121472c9aab4f167d4dadce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2381c45f6121472c9aab4f167d4dadce" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40433 } } }
I20260810 19:22:42.024662 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2381c45f6121472c9aab4f167d4dadce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2381c45f6121472c9aab4f167d4dadce" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 40433 } } }
I20260810 19:22:42.024938 30505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:42.024996 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:42.025852 30510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:42.031474 30510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:42.040940 30510 catalog_manager.cc:1347] Generated new cluster ID: 2f1e030c8e4a49e49aeaa3e6201d429e
I20260810 19:22:42.041057 30510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:22:42.073764 30510 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:22:42.075649 30510 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:22:42.088979 30510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2381c45f6121472c9aab4f167d4dadce: Generated new TSK 0
I20260810 19:22:42.090034 30510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:22:42.135396 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 30398
I20260810 19:22:42.136518 30486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:42.233554 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 30398
2026-08-10T19:22:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2206 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:22:42 dist-test-slave-7lbf krb5kdc[30545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:22:42 dist-test-slave-7lbf krb5kdc[30545](info): set up 2 sockets
Aug 10 19:22:42 dist-test-slave-7lbf krb5kdc[30545](info): commencing operation
krb5kdc: starting...
W20260810 19:22:43.471194 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:22:43 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:22:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:22:43.751053 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44947
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:44.088537 30567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:44.088831 30567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:44.088933 30567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:44.096139 30567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:44.096257 30567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:44.096318 30567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:44.096377 30567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:44.096418 30567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:44.106222 30567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:44.107501 30567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:44.108760 30567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:44.120141 30576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:44.120087 30579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:44.120180 30581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:44.121057 30567 server_base.cc:1034] running on GCE node
I20260810 19:22:44.121891 30567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:44.123328 30567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:44.124527 30567 hybrid_clock.cc:648] HybridClock initialized: now 1786389764124462 us; error 58 us; skew 500 ppm
I20260810 19:22:44.124856 30567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:44 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389764, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:44.127960 30567 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:44.129349 30567 webserver.cc:460] Webserver started at http://127.7.43.62:33381/ using document root <none> and password file <none>
I20260810 19:22:44.130072 30567 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:44.130195 30567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:44.130491 30567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:44.133471 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "a3391a75439e4b4e869d6d9abb592911"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "a6a69892bafe1c97e69aec39fcedcbce"
server_key_iv: "0754bde133f8e7d80cf10c46068567a6"
server_key_version: "encryptionkey@0"
I20260810 19:22:44.134256 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "a3391a75439e4b4e869d6d9abb592911"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "a6a69892bafe1c97e69aec39fcedcbce"
server_key_iv: "0754bde133f8e7d80cf10c46068567a6"
server_key_version: "encryptionkey@0"
I20260810 19:22:44.139237 30567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:22:44.142536 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:44.143807 30567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:22:44.143970 30567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "a3391a75439e4b4e869d6d9abb592911"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "a6a69892bafe1c97e69aec39fcedcbce"
server_key_iv: "0754bde133f8e7d80cf10c46068567a6"
server_key_version: "encryptionkey@0"
I20260810 19:22:44.144141 30567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:44.174620 30567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:44.178548 30567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:44.178813 30567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:44.196223 30567 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:44947
I20260810 19:22:44.196247 30674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:44947 every 8 connection(s)
I20260810 19:22:44.197681 30567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:44.201355 30676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:44.207069 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 30567
I20260810 19:22:44.207254 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:22:44.207870 7340 external_mini_cluster.cc:1428] Setting key 8c8cb2b890d436bdccb0c613d6c7e1e4
I20260810 19:22:44.213426 30676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap starting.
I20260810 19:22:44.217370 30676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:44.218556 30676 log.cc:824] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:44.221608 30676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: No bootstrap required, opened a new log
Aug 10 19:22:44 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:44.231791 30676 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:44.232317 30676 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:44.232432 30676 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3391a75439e4b4e869d6d9abb592911, State: Initialized, Role: FOLLOWER
I20260810 19:22:44.233052 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:44.209896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.7.43.62:44947)
0810 19:22:44.210783 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:44.210799 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:44.210856 (+ 57us) server_negotiation.cc:407] Connection header received
0810 19:22:44.213204 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:44.213231 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:44.213771 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:44.214225 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:44.215607 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:44.217608 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:44.218981 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:44.219804 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:44.219998 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:44.224047 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:44.224059 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:44.224081 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:44.224132 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:44.228256 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:44.228985 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:44.228995 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:44.229008 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:44.229227 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:44.229619 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:44.229628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:44.229631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:44.230453 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:44.230765 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:44.231114 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 19:22:44.231594 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260810 19:22:44.233139 30676 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:44.233428 30676 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:22:44.233630 30676 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:22:44.233816 30676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:44.236354 30676 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:44.236879 30676 leader_election.cc:302] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3391a75439e4b4e869d6d9abb592911; no voters:
I20260810 19:22:44.237435 30676 leader_election.cc:288] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:22:44.237592 30684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:44.238221 30684 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 LEADER]: Becoming Leader. State: Replica: a3391a75439e4b4e869d6d9abb592911, State: Running, Role: LEADER
I20260810 19:22:44.238729 30684 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:44.239449 30676 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:44.241199 30685 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3391a75439e4b4e869d6d9abb592911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } } }
I20260810 19:22:44.241595 30685 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:44.241824 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3391a75439e4b4e869d6d9abb592911. Latest consensus state: current_term: 1 leader_uuid: "a3391a75439e4b4e869d6d9abb592911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } } }
I20260810 19:22:44.242136 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:44.244158 30695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:44.249176 30695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:44.258044 30695 catalog_manager.cc:1347] Generated new cluster ID: 5c73e65d64214116b945157d363ba706
I20260810 19:22:44.258154 30695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:22:44.276707 30695 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:22:44.278091 30695 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:22:44.291353 30695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Generated new TSK 0
I20260810 19:22:44.292196 30695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:22:44.364470 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:44947
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:44.697985 30719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:44.698302 30719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:44.698417 30719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:44.705892 30719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:44.706015 30719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:44.706151 30719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:22:44.716313 30719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:44.717656 30719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:44.719017 30719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:44.732023 30728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:44.731989 30726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:44.732604 30731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:44.733486 30719 server_base.cc:1034] running on GCE node
I20260810 19:22:44.734153 30719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:44.735085 30719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:44.736281 30719 hybrid_clock.cc:648] HybridClock initialized: now 1786389764736212 us; error 61 us; skew 500 ppm
I20260810 19:22:44.736581 30719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:44 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389764, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:44.739609 30719 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:22:44.741050 30719 webserver.cc:460] Webserver started at http://127.7.43.1:35053/ using document root <none> and password file <none>
I20260810 19:22:44.741716 30719 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:44.741827 30719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:44.742132 30719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:44.745205 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "01e87851796246ad873d524198316072"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "e0bf29350a02a5cee7d5f8d954e8038a"
server_key_iv: "e217ed454d9801b444655277c36f465f"
server_key_version: "encryptionkey@0"
I20260810 19:22:44.745947 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "01e87851796246ad873d524198316072"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "e0bf29350a02a5cee7d5f8d954e8038a"
server_key_iv: "e217ed454d9801b444655277c36f465f"
server_key_version: "encryptionkey@0"
I20260810 19:22:44.751247 30719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 19:22:44.754496 30745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:44.755736 30719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:22:44.755867 30719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "01e87851796246ad873d524198316072"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "e0bf29350a02a5cee7d5f8d954e8038a"
server_key_iv: "e217ed454d9801b444655277c36f465f"
server_key_version: "encryptionkey@0"
I20260810 19:22:44.756052 30719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:44.770263 30719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:44.774022 30719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:44.774287 30719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:44.775229 30719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:44.776542 30719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:44.776615 30719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:44.776733 30719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:44.776786 30719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:44.811105 30719 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:40111
I20260810 19:22:44.811141 30930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:40111 every 8 connection(s)
I20260810 19:22:44.812356 30719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:22:44.822501 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 30719
I20260810 19:22:44.822700 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:22:44.823401 7340 external_mini_cluster.cc:1428] Setting key ca95031f20288fe4cdffd2f37ec229a0
Aug 10 19:22:44 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389764, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:44.831871 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:44.814909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:39287 (local address 127.7.43.62:44947)
0810 19:22:44.815365 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:44.815370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:44.816295 (+ 925us) server_negotiation.cc:407] Connection header received
0810 19:22:44.818073 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:44.818078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:44.818143 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:44.818234 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:44.819595 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:44.820317 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:44.821816 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:44.822214 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:44.822327 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:44.826058 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:44.826065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:44.826070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:44.826108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:44.828925 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:44.829624 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:44.829630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:44.829633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:44.829777 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:44.830171 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:44.830175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:44.830177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:44.830496 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:44.830662 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:44.831314 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 19:22:44.831535 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
I20260810 19:22:44.832639 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:44.815446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:39287)
0810 19:22:44.816139 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:44.816180 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:44.817840 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:44.818419 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:44.818427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:44.818919 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:44.819465 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:44.819483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:44.820443 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:44.820447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:44.821704 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:44.821717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:44.822441 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:44.822448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:44.822703 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:44.823357 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:44.823379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:44.825823 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:44.829053 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:44.829060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:44.829072 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:44.829478 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:44.829866 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:44.829870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:44.829872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:44.830067 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:44.830620 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:44.830625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:44.830947 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 19:22:44.831172 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 19:22:44.831546 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":260,"threads_started":1}
I20260810 19:22:44.834126 30934 heartbeater.cc:347] Connected to a master server at 127.7.43.62:44947
I20260810 19:22:44.834414 30934 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:44.835135 30934 heartbeater.cc:511] Master 127.7.43.62:44947 requested a full tablet report, sending...
I20260810 19:22:44.837160 30620 ts_manager.cc:194] Registered new tserver with Master: 01e87851796246ad873d524198316072 (127.7.43.1:40111)
I20260810 19:22:44.838620 30620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:39287
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:22:44.884956 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:44947
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:45.220711 30942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:45.221057 30942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:45.221220 30942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:45.228665 30942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:45.228824 30942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:45.229020 30942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:22:45.239073 30942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:45.240449 30942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:45.241869 30942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:45.255254 30959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:45.255290 30952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:45.255508 30956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:45.256127 30942 server_base.cc:1034] running on GCE node
I20260810 19:22:45.256749 30942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:45.257696 30942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:45.258894 30942 hybrid_clock.cc:648] HybridClock initialized: now 1786389765258842 us; error 48 us; skew 500 ppm
I20260810 19:22:45.259222 30942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:45 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389765, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:45.266865 30942 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:22:45.268296 30942 webserver.cc:460] Webserver started at http://127.7.43.2:42581/ using document root <none> and password file <none>
I20260810 19:22:45.269049 30942 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:45.269166 30942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:45.269462 30942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:45.272406 30942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "8e57ecfa88b047ccade60ddde7473261"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "d038b3acfefa32528f7a8592ca5e4c8c"
server_key_iv: "a21d7b98da6516510ffb29f0315131a8"
server_key_version: "encryptionkey@0"
I20260810 19:22:45.273156 30942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "8e57ecfa88b047ccade60ddde7473261"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "d038b3acfefa32528f7a8592ca5e4c8c"
server_key_iv: "a21d7b98da6516510ffb29f0315131a8"
server_key_version: "encryptionkey@0"
I20260810 19:22:45.278391 30942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 19:22:45.282070 30971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:45.283427 30942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:22:45.283598 30942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "8e57ecfa88b047ccade60ddde7473261"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "d038b3acfefa32528f7a8592ca5e4c8c"
server_key_iv: "a21d7b98da6516510ffb29f0315131a8"
server_key_version: "encryptionkey@0"
I20260810 19:22:45.283810 30942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:45.300763 30942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:45.304639 30942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:45.304939 30942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:45.305997 30942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:45.307461 30942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:45.307550 30942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:45.307687 30942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:45.307750 30942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:45.340014 30942 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45655
I20260810 19:22:45.340046 31143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45655 every 8 connection(s)
I20260810 19:22:45.341452 30942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:22:45.350571 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 30942
I20260810 19:22:45.350759 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:22:45.351341 7340 external_mini_cluster.cc:1428] Setting key fa129986d4d01878a550afb8e07466a6
Aug 10 19:22:45 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389765, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:45.361759 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.344105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:35957 (local address 127.7.43.62:44947)
0810 19:22:45.344518 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:45.344522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:45.345614 (+ 1092us) server_negotiation.cc:407] Connection header received
0810 19:22:45.347395 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:45.347400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:45.347505 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:45.347615 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:45.349093 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.349855 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.352027 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.352532 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.352653 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.356263 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:45.356268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:45.356271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:45.356302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:45.358738 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:45.359499 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:45.359503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:45.359505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:45.359629 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:45.360030 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:45.360034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:45.360036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:45.360357 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:45.360503 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:45.361157 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 19:22:45.361380 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":208,"threads_started":1}
I20260810 19:22:45.362636 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.344687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:35957)
0810 19:22:45.345444 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:45.345488 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:45.347137 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:45.347842 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:45.347849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:45.348326 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:45.348924 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.348944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.350006 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.350009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.351878 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.351892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.352730 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.352736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.352938 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.353654 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:45.353684 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:45.356058 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:45.358877 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:45.358886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:45.358901 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:45.359375 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:45.359733 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:45.359738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:45.359742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:45.359925 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:45.360481 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:45.360489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:45.360799 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 19:22:45.361042 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 19:22:45.361569 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":305,"threads_started":1}
I20260810 19:22:45.364001 31144 heartbeater.cc:347] Connected to a master server at 127.7.43.62:44947
I20260810 19:22:45.364271 31144 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:45.364965 31144 heartbeater.cc:511] Master 127.7.43.62:44947 requested a full tablet report, sending...
I20260810 19:22:45.366322 30620 ts_manager.cc:194] Registered new tserver with Master: 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2:45655)
I20260810 19:22:45.367091 30620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:35957
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:22:45.410651 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:44947
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:45.751178 31157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:45.751467 31157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:45.751614 31157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:45.758920 31157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:45.759044 31157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:45.759214 31157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:22:45.769359 31157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:45.770838 31157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:45.772248 31157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:45.785882 31163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:45.785935 31164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:45.786278 31167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:45.787004 31157 server_base.cc:1034] running on GCE node
I20260810 19:22:45.787693 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:45.788700 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:45.789937 31157 hybrid_clock.cc:648] HybridClock initialized: now 1786389765789888 us; error 39 us; skew 500 ppm
I20260810 19:22:45.790277 31157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:45 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389765, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:45.793296 31157 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:22:45.794667 31157 webserver.cc:460] Webserver started at http://127.7.43.3:35601/ using document root <none> and password file <none>
I20260810 19:22:45.795328 31157 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:45.795437 31157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:45.795728 31157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:22:45.798638 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "3322971d62e14886a8a95f953ba3e77f"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "5284120fd679b9a06cc30b389a7f642c"
server_key_iv: "de18820e95d2940fa471c5bd45148152"
server_key_version: "encryptionkey@0"
I20260810 19:22:45.799351 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "3322971d62e14886a8a95f953ba3e77f"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "5284120fd679b9a06cc30b389a7f642c"
server_key_iv: "de18820e95d2940fa471c5bd45148152"
server_key_version: "encryptionkey@0"
I20260810 19:22:45.804278 31157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:22:45.807482 31180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:45.808701 31157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:22:45.808847 31157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "3322971d62e14886a8a95f953ba3e77f"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "5284120fd679b9a06cc30b389a7f642c"
server_key_iv: "de18820e95d2940fa471c5bd45148152"
server_key_version: "encryptionkey@0"
I20260810 19:22:45.809022 31157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:45.834182 31157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:45.838295 31157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:45.838589 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:45.839617 31157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:45.841042 31157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:45.841127 31157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:45.841250 31157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:45.841305 31157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:45.841702 30934 heartbeater.cc:503] Master 127.7.43.62:44947 was elected leader, sending a full tablet report...
I20260810 19:22:45.872247 31157 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:40499
I20260810 19:22:45.872298 31353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:40499 every 8 connection(s)
I20260810 19:22:45.873508 31157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:22:45.877758 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 31157
I20260810 19:22:45.877930 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:22:45.878353 7340 external_mini_cluster.cc:1428] Setting key 78ae3825fc53938a46e92112b0554e06
Aug 10 19:22:45 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389765, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:45.893047 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.876044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:33031 (local address 127.7.43.62:44947)
0810 19:22:45.876468 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:45.876473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:45.877377 (+ 904us) server_negotiation.cc:407] Connection header received
0810 19:22:45.879418 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:45.879429 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:45.879536 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:45.879628 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:45.881586 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.882414 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.883890 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.884306 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.884394 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.887819 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:45.887824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:45.887828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:45.887861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:45.890265 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:45.890931 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:45.890934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:45.890937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:45.891055 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:45.891376 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:45.891379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:45.891381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:45.891626 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:45.891744 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:45.892492 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 19:22:45.892688 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":205,"threads_started":1}
I20260810 19:22:45.894198 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.876516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:33031)
0810 19:22:45.877216 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:45.877254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:45.879149 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:45.879925 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:45.879948 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:45.880670 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:45.881402 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.881421 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.882552 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.882556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.883781 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.883793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.884484 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.884489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.884681 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.885300 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:45.885323 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:45.887629 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:45.890394 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:45.890401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:45.890413 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:45.890837 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:45.891126 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:45.891129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:45.891131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:45.891299 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:45.891717 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:45.891722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:45.892080 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 19:22:45.892374 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 19:22:45.892850 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":269,"threads_started":1}
I20260810 19:22:45.895473 31356 heartbeater.cc:347] Connected to a master server at 127.7.43.62:44947
I20260810 19:22:45.895762 31356 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:45.896459 31356 heartbeater.cc:511] Master 127.7.43.62:44947 requested a full tablet report, sending...
I20260810 19:22:45.897815 30620 ts_manager.cc:194] Registered new tserver with Master: 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499)
I20260810 19:22:45.898598 30620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:33031
I20260810 19:22:45.906394 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:22:45.922631 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.910902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.7.43.62:44947)
0810 19:22:45.911135 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:45.911142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:45.911484 (+ 342us) server_negotiation.cc:407] Connection header received
0810 19:22:45.911611 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:45.911614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:45.911690 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:45.911768 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:45.913158 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.913948 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.915190 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.915567 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.915651 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.918036 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:45.918041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:45.918045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:45.918075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:45.920443 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:45.921038 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:45.921041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:45.921044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:45.921155 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:45.921421 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:45.921424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:45.921426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:45.921770 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:45.921952 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:45.922195 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 19:22:45.922365 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:22:45.926318 30620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:22:45.933084 30620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:22:45.955540 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.944757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:39318)
0810 19:22:45.945795 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:45.945844 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:45.946036 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:45.947085 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:45.947092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:45.947122 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:45.947566 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.947580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.949705 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.949711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.952177 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.952194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.954481 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.954489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.954725 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.954778 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 19:22:45.954911 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 19:22:45.955115 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":411,"threads_started":1}
I20260810 19:22:45.956465 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.944799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.7.43.1:40111)
0810 19:22:45.945441 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:45.945450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:45.946237 (+ 787us) server_negotiation.cc:407] Connection header received
0810 19:22:45.946345 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:45.946353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:45.946629 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:45.946812 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:45.947776 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.949501 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.952370 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.953964 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.954088 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.954339 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:45.955944 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0810 19:22:45.956132 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":872,"threads_started":1}
I20260810 19:22:45.961074 30827 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4e5d1746ec45c182d94fb1d91788c1 (DEFAULT_TABLE table=test-table [id=f38e0e663e3046108ba1f09709ac8103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:45.962627 30827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4e5d1746ec45c182d94fb1d91788c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:45.967025 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.946463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45655 (local address 127.0.0.1:41368)
0810 19:22:45.957211 (+ 10748us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:45.957230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:45.957327 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:45.958508 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:45.958513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:45.958539 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:45.958751 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.958762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.962533 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.962538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.964321 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.964347 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.966236 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.966242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.966418 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.966463 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:22:45.966558 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:22:45.966698 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10643,"thread_start_us":13736,"threads_started":1}
I20260810 19:22:45.967832 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.947039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.7.43.2:45655)
0810 19:22:45.957801 (+ 10762us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:45.957810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:45.957838 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:22:45.957939 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:45.957946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:45.958194 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:45.960833 (+ 2639us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:45.961082 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.962382 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.964530 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.966055 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.966203 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.967052 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:45.967131 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 19:22:45.967531 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10599,"thread_start_us":13492,"threads_started":1}
I20260810 19:22:45.968947 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.945216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:40499 (local address 127.0.0.1:53286)
0810 19:22:45.946181 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:45.946196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:45.946291 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:45.962713 (+ 16422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:45.962717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:45.962745 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:45.963144 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.963156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.965051 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.965061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.966550 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:45.966562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.968178 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.968184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:45.968339 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.968362 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:22:45.968451 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 19:22:45.968576 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":1,"thread_start_us":699,"threads_started":1}
I20260810 19:22:45.968989 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:45.945800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.7.43.3:40499)
0810 19:22:45.962137 (+ 16337us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:45.962145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:45.962179 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:22:45.962281 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:45.962289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:45.962520 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:45.963317 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:45.963623 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.964925 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.966715 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:45.967897 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:45.968026 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:45.968236 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:45.968506 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 19:22:45.968652 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16124,"thread_start_us":15928,"threads_started":1}
I20260810 19:22:45.971736 31040 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4e5d1746ec45c182d94fb1d91788c1 (DEFAULT_TABLE table=test-table [id=f38e0e663e3046108ba1f09709ac8103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:45.972275 31250 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4e5d1746ec45c182d94fb1d91788c1 (DEFAULT_TABLE table=test-table [id=f38e0e663e3046108ba1f09709ac8103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:45.973297 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4e5d1746ec45c182d94fb1d91788c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:45.973412 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4e5d1746ec45c182d94fb1d91788c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:45.979175 31395 tablet_bootstrap.cc:492] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Bootstrap starting.
I20260810 19:22:45.983685 31395 tablet_bootstrap.cc:654] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:45.984993 31395 log.cc:824] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:45.986071 31396 tablet_bootstrap.cc:492] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Bootstrap starting.
I20260810 19:22:45.990643 31396 tablet_bootstrap.cc:654] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:45.991981 31396 log.cc:824] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:45.992033 31398 tablet_bootstrap.cc:492] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Bootstrap starting.
I20260810 19:22:45.995544 31396 tablet_bootstrap.cc:492] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: No bootstrap required, opened a new log
I20260810 19:22:45.995676 31398 tablet_bootstrap.cc:654] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:45.995832 31396 ts_tablet_manager.cc:1397] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 19:22:45.997082 31398 log.cc:824] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:46.000535 31395 tablet_bootstrap.cc:492] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: No bootstrap required, opened a new log
I20260810 19:22:46.000838 31395 ts_tablet_manager.cc:1397] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.019s
I20260810 19:22:46.003398 31398 tablet_bootstrap.cc:492] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: No bootstrap required, opened a new log
I20260810 19:22:46.003690 31398 ts_tablet_manager.cc:1397] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 19:22:46.006669 31396 raft_consensus.cc:348] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.007136 31396 raft_consensus.cc:374] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:46.007239 31396 raft_consensus.cc:729] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3322971d62e14886a8a95f953ba3e77f, State: Initialized, Role: FOLLOWER
I20260810 19:22:46.007921 31396 consensus_queue.cc:260] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.009115 31356 heartbeater.cc:503] Master 127.7.43.62:44947 was elected leader, sending a full tablet report...
I20260810 19:22:46.009756 31396 ts_tablet_manager.cc:1428] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
I20260810 19:22:46.010157 31395 raft_consensus.cc:348] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.010602 31395 raft_consensus.cc:374] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:46.010704 31395 raft_consensus.cc:729] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e87851796246ad873d524198316072, State: Initialized, Role: FOLLOWER
I20260810 19:22:46.011257 31395 consensus_queue.cc:260] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.012858 31395 ts_tablet_manager.cc:1428] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Time spent starting tablet: real 0.012s user 0.004s sys 0.005s
I20260810 19:22:46.014726 31398 raft_consensus.cc:348] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.015166 31398 raft_consensus.cc:374] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:46.015272 31398 raft_consensus.cc:729] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e57ecfa88b047ccade60ddde7473261, State: Initialized, Role: FOLLOWER
I20260810 19:22:46.015905 31398 consensus_queue.cc:260] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.017004 31144 heartbeater.cc:503] Master 127.7.43.62:44947 was elected leader, sending a full tablet report...
I20260810 19:22:46.017534 31398 ts_tablet_manager.cc:1428] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260810 19:22:46.064539 30935 tablet.cc:2369] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:22:46.093541 31145 tablet.cc:2369] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:22:46.125121 31357 tablet.cc:2369] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:22:46.368724 31404 raft_consensus.cc:482] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:22:46.369028 31404 raft_consensus.cc:504] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.370747 31404 leader_election.cc:288] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e87851796246ad873d524198316072 (127.7.43.1:40111), 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499)
I20260810 19:22:46.377839 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.371245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:48955 (local address 127.7.43.1:40111)
0810 19:22:46.371445 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:46.371450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:46.372331 (+ 881us) server_negotiation.cc:407] Connection header received
0810 19:22:46.372521 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:46.372527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:46.372588 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:46.372660 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:46.373307 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.374056 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.376191 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.376968 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.377091 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.377195 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:46.377538 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 19:22:46.377657 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:22:46.378141 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.371170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.7.43.2:48955)
0810 19:22:46.372237 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:46.372255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:46.372382 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:46.372696 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:46.372701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:46.372740 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:46.373158 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:46.373168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.374189 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.374195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:46.376047 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:46.376060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.377176 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.377182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:46.377348 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.377382 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:22:46.377480 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:22:46.377629 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":102,"thread_start_us":472,"threads_started":1}
I20260810 19:22:46.379287 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.371169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:56913 (local address 127.7.43.3:40499)
0810 19:22:46.371348 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:46.371355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:46.372243 (+ 888us) server_negotiation.cc:407] Connection header received
0810 19:22:46.372418 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:46.372424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:46.372493 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:46.372567 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:46.373143 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.373909 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.377279 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.378458 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.378599 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.378695 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:46.378921 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 19:22:46.379050 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:22:46.379366 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.371169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:40499 (local address 127.7.43.2:56913)
0810 19:22:46.372131 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:46.372146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:46.372267 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:46.372594 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:46.372598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:46.372622 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:46.373003 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:46.373017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.374042 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.374046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:46.376031 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:46.376047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.378646 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.378651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:46.378766 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.378790 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:22:46.378872 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:22:46.379009 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":71,"thread_start_us":538,"threads_started":1}
I20260810 19:22:46.379206 30851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5d1746ec45c182d94fb1d91788c1" candidate_uuid: "8e57ecfa88b047ccade60ddde7473261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e87851796246ad873d524198316072" is_pre_election: true
I20260810 19:22:46.379745 30851 raft_consensus.cc:2454] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e57ecfa88b047ccade60ddde7473261 in term 0.
I20260810 19:22:46.380251 31285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5d1746ec45c182d94fb1d91788c1" candidate_uuid: "8e57ecfa88b047ccade60ddde7473261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3322971d62e14886a8a95f953ba3e77f" is_pre_election: true
I20260810 19:22:46.380410 30975 leader_election.cc:302] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e87851796246ad873d524198316072, 8e57ecfa88b047ccade60ddde7473261; no voters:
I20260810 19:22:46.380795 31285 raft_consensus.cc:2454] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e57ecfa88b047ccade60ddde7473261 in term 0.
I20260810 19:22:46.380846 31404 raft_consensus.cc:2781] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:22:46.380985 31404 raft_consensus.cc:482] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:22:46.381088 31404 raft_consensus.cc:3037] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:46.383392 31404 raft_consensus.cc:504] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.383975 31404 leader_election.cc:288] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [CANDIDATE]: Term 1 election: Requested vote from peers 01e87851796246ad873d524198316072 (127.7.43.1:40111), 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499)
I20260810 19:22:46.384532 31285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5d1746ec45c182d94fb1d91788c1" candidate_uuid: "8e57ecfa88b047ccade60ddde7473261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3322971d62e14886a8a95f953ba3e77f"
I20260810 19:22:46.384531 30851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4e5d1746ec45c182d94fb1d91788c1" candidate_uuid: "8e57ecfa88b047ccade60ddde7473261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e87851796246ad873d524198316072"
I20260810 19:22:46.384720 31285 raft_consensus.cc:3037] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:46.384716 30851 raft_consensus.cc:3037] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:46.386901 30851 raft_consensus.cc:2454] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e57ecfa88b047ccade60ddde7473261 in term 1.
I20260810 19:22:46.386925 31285 raft_consensus.cc:2454] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e57ecfa88b047ccade60ddde7473261 in term 1.
I20260810 19:22:46.387295 30975 leader_election.cc:302] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e87851796246ad873d524198316072, 8e57ecfa88b047ccade60ddde7473261; no voters:
I20260810 19:22:46.387605 31404 raft_consensus.cc:2781] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:46.388041 31404 raft_consensus.cc:686] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 1 LEADER]: Becoming Leader. State: Replica: 8e57ecfa88b047ccade60ddde7473261, State: Running, Role: LEADER
I20260810 19:22:46.388751 31404 consensus_queue.cc:237] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } }
I20260810 19:22:46.393565 30620 catalog_manager.cc:5495] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2). New cstate: current_term: 1 leader_uuid: "8e57ecfa88b047ccade60ddde7473261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } health_report { overall_health: HEALTHY } } }
I20260810 19:22:46.424984 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.419554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.7.43.2:45655)
0810 19:22:46.419744 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:46.419749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:46.419768 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:22:46.419998 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:46.420003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:46.420083 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:46.420212 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:22:46.420606 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.421331 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.422811 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.423318 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.423426 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.423893 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:22:46.424396 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:22:46.424538 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:46.424627 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 19:22:46.424709 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:22:46.433317 30851 raft_consensus.cc:1261] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Refusing update from remote peer 8e57ecfa88b047ccade60ddde7473261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:46.433313 31285 raft_consensus.cc:1261] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Refusing update from remote peer 8e57ecfa88b047ccade60ddde7473261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:46.434361 31404 consensus_queue.cc:1035] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:46.434659 31409 consensus_queue.cc:1035] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:46.460685 30617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:22:46.461023 30617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:22:46.464824 30617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Sending DeleteTablet for 3 replicas of tablet 2f4e5d1746ec45c182d94fb1d91788c1
I20260810 19:22:46.466176 30827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4e5d1746ec45c182d94fb1d91788c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:46 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:39318
I20260810 19:22:46.466234 31250 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4e5d1746ec45c182d94fb1d91788c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:46 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:53286
I20260810 19:22:46.466404 31040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4e5d1746ec45c182d94fb1d91788c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:46 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:41368
I20260810 19:22:46.467463 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 30719
I20260810 19:22:46.477440 31433 tablet_replica.cc:330] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: stopping tablet replica
I20260810 19:22:46.478320 31433 raft_consensus.cc:2229] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:22:46.479192 31430 tablet_replica.cc:330] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: stopping tablet replica
I20260810 19:22:46.480077 31433 raft_consensus.cc:2258] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:46.480237 31431 tablet_replica.cc:330] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: stopping tablet replica
I20260810 19:22:46.480724 31431 raft_consensus.cc:2229] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:22:46.480716 31430 raft_consensus.cc:2229] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:22:46.481310 31431 raft_consensus.cc:2258] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:46.481957 31430 raft_consensus.cc:2258] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:46.483656 31431 ts_tablet_manager.cc:1905] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:46.483952 31433 ts_tablet_manager.cc:1905] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:46.487553 31430 ts_tablet_manager.cc:1905] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:46.491040 31431 ts_tablet_manager.cc:1918] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:46.491278 31431 log.cc:1196] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/2f4e5d1746ec45c182d94fb1d91788c1
Aug 10 19:22:46 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:22:46.491891 31431 ts_tablet_manager.cc:1939] T 2f4e5d1746ec45c182d94fb1d91788c1 P 3322971d62e14886a8a95f953ba3e77f: Deleting consensus metadata
I20260810 19:22:46.494259 30594 catalog_manager.cc:4841] TS 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499): tablet 2f4e5d1746ec45c182d94fb1d91788c1 (table test-table [id=f38e0e663e3046108ba1f09709ac8103]) successfully deleted
I20260810 19:22:46.496785 31433 ts_tablet_manager.cc:1918] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:46.497103 31433 log.cc:1196] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/2f4e5d1746ec45c182d94fb1d91788c1
I20260810 19:22:46.497849 31433 ts_tablet_manager.cc:1939] T 2f4e5d1746ec45c182d94fb1d91788c1 P 8e57ecfa88b047ccade60ddde7473261: Deleting consensus metadata
I20260810 19:22:46.500002 30594 catalog_manager.cc:4841] TS 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2:45655): tablet 2f4e5d1746ec45c182d94fb1d91788c1 (table test-table [id=f38e0e663e3046108ba1f09709ac8103]) successfully deleted
I20260810 19:22:46.500696 31430 ts_tablet_manager.cc:1918] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:46.500866 31430 log.cc:1196] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/2f4e5d1746ec45c182d94fb1d91788c1
I20260810 19:22:46.501405 31430 ts_tablet_manager.cc:1939] T 2f4e5d1746ec45c182d94fb1d91788c1 P 01e87851796246ad873d524198316072: Deleting consensus metadata
I20260810 19:22:46.502058 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.468715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.7.43.1:40111)
0810 19:22:46.468898 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:46.468905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:46.479511 (+ 10606us) server_negotiation.cc:407] Connection header received
0810 19:22:46.479755 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:46.479762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:46.479856 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:46.479993 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:46.482493 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.483658 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.485746 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.486418 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.486529 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.492934 (+ 6405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:46.492954 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:46.492973 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:46.493032 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:46.497904 (+ 4872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:46.498951 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:46.498962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:46.498972 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:46.499168 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:46.499739 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:46.499745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:46.499748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:46.500310 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:46.500612 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:46.501265 (+ 653us) server_negotiation.cc:299] Negotiation successful
0810 19:22:46.501595 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:22:46.502753 30921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:46.503510 30597 catalog_manager.cc:4841] TS 01e87851796246ad873d524198316072 (127.7.43.1:40111): tablet 2f4e5d1746ec45c182d94fb1d91788c1 (table test-table [id=f38e0e663e3046108ba1f09709ac8103]) successfully deleted
I20260810 19:22:46.594476 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 30719
I20260810 19:22:46.620278 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 30942
Aug 10 19:22:46 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:22:46.638531 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.621117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.7.43.2:45655)
0810 19:22:46.621293 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:46.621299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:46.621323 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:22:46.621566 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:46.621571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:46.621642 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:46.621740 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:46.623375 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.624271 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.626193 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.626666 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.626776 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.632054 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:46.632073 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:46.632090 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:46.632143 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:46.635598 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:46.636421 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:46.636429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:46.636432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:46.636577 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:46.637014 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:46.637019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:46.637021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:46.637431 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:46.637775 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:46.638023 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 19:22:46.638197 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:22:46.639273 31137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:46.752969 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 30942
I20260810 19:22:46.775597 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 31157
Aug 10 19:22:46 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:22:46.796298 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:46.776453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.7.43.3:40499)
0810 19:22:46.776647 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:46.776653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:46.776672 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:22:46.776801 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:46.776805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:46.776872 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:46.776965 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:46.778322 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.779040 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.780309 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:46.780712 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:46.780820 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:46.785130 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:46.785144 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:46.785161 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:46.785194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:46.788418 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:46.789180 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:46.789190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:46.789193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:46.789374 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:46.789781 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:46.789784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:46.789786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:46.790217 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:46.790512 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:46.795777 (+ 5265us) server_negotiation.cc:299] Negotiation successful
0810 19:22:46.795976 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:22:46.797200 31349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:46.883770 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 31157
I20260810 19:22:46.907449 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 30567
I20260810 19:22:46.908633 30669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:47.015743 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 30567
I20260810 19:22:47.043431 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44947
--webserver_interface=127.7.43.62
--webserver_port=33381
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:47.376398 31447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:47.376703 31447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:47.376833 31447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:47.384160 31447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:47.384289 31447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:47.384363 31447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:47.384443 31447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:47.384508 31447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:47.394627 31447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=33381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:47.395992 31447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:47.397364 31447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:47.423051 31460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:47.423074 31456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:47.423018 31457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:47.429210 31447 server_base.cc:1034] running on GCE node
I20260810 19:22:47.430253 31447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:47.431941 31447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:47.433184 31447 hybrid_clock.cc:648] HybridClock initialized: now 1786389767433117 us; error 47 us; skew 500 ppm
I20260810 19:22:47.433660 31447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:47 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389767, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:47.437417 31447 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:47.439388 31447 webserver.cc:460] Webserver started at http://127.7.43.62:33381/ using document root <none> and password file <none>
I20260810 19:22:47.440243 31447 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:47.440402 31447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:47.446856 31447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:22:47.449957 31478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:47.451493 31447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:22:47.451686 31447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "a3391a75439e4b4e869d6d9abb592911"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "a6a69892bafe1c97e69aec39fcedcbce"
server_key_iv: "0754bde133f8e7d80cf10c46068567a6"
server_key_version: "encryptionkey@0"
I20260810 19:22:47.452785 31447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:47.474252 31447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:47.478572 31447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:47.478856 31447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:47.497138 31447 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:44947
I20260810 19:22:47.497159 31557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:44947 every 8 connection(s)
I20260810 19:22:47.498735 31447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:47.499274 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 31447
I20260810 19:22:47.500211 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:40111
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:44947
--webserver_port=35053
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:47.506210 31558 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:22:47.510179 31558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap starting.
I20260810 19:22:47.530280 31558 log.cc:824] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:47.543114 31558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:22:47.543656 31558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap complete.
I20260810 19:22:47.553095 31558 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:47.554029 31558 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3391a75439e4b4e869d6d9abb592911, State: Initialized, Role: FOLLOWER
I20260810 19:22:47.554708 31558 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:47.554952 31558 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:22:47.555073 31558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:22:47.555243 31558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:22:47.559435 31558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:47.559942 31558 leader_election.cc:302] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3391a75439e4b4e869d6d9abb592911; no voters:
I20260810 19:22:47.560546 31558 leader_election.cc:288] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:22:47.560724 31566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:22:47.561712 31566 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 LEADER]: Becoming Leader. State: Replica: a3391a75439e4b4e869d6d9abb592911, State: Running, Role: LEADER
I20260810 19:22:47.562332 31566 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:47.562633 31558 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:47.565030 31570 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3391a75439e4b4e869d6d9abb592911. Latest consensus state: current_term: 2 leader_uuid: "a3391a75439e4b4e869d6d9abb592911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } } }
I20260810 19:22:47.565644 31570 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:47.565055 31569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3391a75439e4b4e869d6d9abb592911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } } }
I20260810 19:22:47.569864 31569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:47.570694 31575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:47.579352 31575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:47.580838 31575 catalog_manager.cc:1259] Loaded cluster ID: 5c73e65d64214116b945157d363ba706
I20260810 19:22:47.581275 31575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:22:47.584527 31575 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 19:22:47.601243 31575 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 19:22:47.626576 31575 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:22:47.628551 31575 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Loaded TSK: 0
I20260810 19:22:47.630156 31575 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:22:47.877612 31560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:47.877905 31560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:47.878038 31560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:47.885308 31560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:47.885427 31560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:47.885612 31560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:22:47.895728 31560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:40111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=35053
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:47.897035 31560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:47.898466 31560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:47.913240 31607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:47.913314 31610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:47.913345 31608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:47.914302 31560 server_base.cc:1034] running on GCE node
I20260810 19:22:47.915089 31560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:47.916309 31560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:47.917593 31560 hybrid_clock.cc:648] HybridClock initialized: now 1786389767917500 us; error 74 us; skew 500 ppm
I20260810 19:22:47.917994 31560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:47 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389767, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:47.921913 31560 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:22:47.923601 31560 webserver.cc:460] Webserver started at http://127.7.43.1:35053/ using document root <none> and password file <none>
I20260810 19:22:47.924552 31560 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:47.924723 31560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:47.932072 31560 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:22:47.935470 31625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:47.937021 31560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:22:47.937184 31560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "01e87851796246ad873d524198316072"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "e0bf29350a02a5cee7d5f8d954e8038a"
server_key_iv: "e217ed454d9801b444655277c36f465f"
server_key_version: "encryptionkey@0"
I20260810 19:22:47.938596 31560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:47.968668 31560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:47.973891 31560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:47.974176 31560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:47.975383 31560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:47.977142 31560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:47.977258 31560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:47.977362 31560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:47.977432 31560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.012221 31560 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:40111
I20260810 19:22:48.012274 31806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:40111 every 8 connection(s)
I20260810 19:22:48.013833 31560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:22:48.019338 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 31560
I20260810 19:22:48.019948 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:45655
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:44947
--webserver_port=42581
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:22:48 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389767, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:48.038306 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.016741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:45795 (local address 127.7.43.62:44947)
0810 19:22:48.017439 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:48.017456 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:48.018063 (+ 607us) server_negotiation.cc:407] Connection header received
0810 19:22:48.020397 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:48.020416 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:48.020821 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:48.021222 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:48.022786 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.024555 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.026239 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.027132 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.027294 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.030856 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:48.030866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:48.030884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:48.030934 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:48.034017 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.034894 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.034901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.034907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.035080 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.035527 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.035530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.035532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.035976 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:48.036133 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:48.036988 (+ 855us) server_negotiation.cc:299] Negotiation successful
0810 19:22:48.037320 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
I20260810 19:22:48.038623 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.016925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:45795)
0810 19:22:48.017860 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:48.017919 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:48.019805 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:48.021372 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:48.021380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:48.021954 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:48.022554 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:48.022576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.024731 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.024736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:48.026089 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:48.026104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.027370 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.027376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:48.027585 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.028254 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:48.028278 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:48.030566 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:48.034168 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:48.034177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:48.034192 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:48.034763 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:48.035191 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:48.035195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:48.035199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:48.035426 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:48.036119 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:48.036126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:48.036530 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 19:22:48.036814 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 19:22:48.037339 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":357,"threads_started":1}
I20260810 19:22:48.041404 31807 heartbeater.cc:347] Connected to a master server at 127.7.43.62:44947
I20260810 19:22:48.041715 31807 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:48.042407 31807 heartbeater.cc:511] Master 127.7.43.62:44947 requested a full tablet report, sending...
I20260810 19:22:48.044896 31507 ts_manager.cc:194] Registered new tserver with Master: 01e87851796246ad873d524198316072 (127.7.43.1:40111)
I20260810 19:22:48.046945 31507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:45795
W20260810 19:22:48.350348 31814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:48.350632 31814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:48.350736 31814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:48.357762 31814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:48.357864 31814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:48.357990 31814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:22:48.367342 31814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:45655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=42581
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:48.368541 31814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:48.369833 31814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:48.382267 31822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:48.382334 31827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:48.382302 31823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:48.383548 31814 server_base.cc:1034] running on GCE node
I20260810 19:22:48.384148 31814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:48.385051 31814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:48.386232 31814 hybrid_clock.cc:648] HybridClock initialized: now 1786389768386176 us; error 45 us; skew 500 ppm
I20260810 19:22:48.386531 31814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:48 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389768, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:48.389621 31814 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:22:48.391072 31814 webserver.cc:460] Webserver started at http://127.7.43.2:42581/ using document root <none> and password file <none>
I20260810 19:22:48.391753 31814 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:48.391894 31814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:48.397279 31814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:22:48.399971 31836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.401260 31814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:22:48.401409 31814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "8e57ecfa88b047ccade60ddde7473261"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "d038b3acfefa32528f7a8592ca5e4c8c"
server_key_iv: "a21d7b98da6516510ffb29f0315131a8"
server_key_version: "encryptionkey@0"
I20260810 19:22:48.402437 31814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:48.437201 31814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:48.441176 31814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:48.441427 31814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:48.442404 31814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:48.443766 31814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:48.443862 31814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.443974 31814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:48.444043 31814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.474766 31814 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45655
I20260810 19:22:48.474824 32021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45655 every 8 connection(s)
I20260810 19:22:48.476229 31814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:22:48.479668 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 31814
I20260810 19:22:48.480284 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:40499
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:44947
--webserver_port=35601
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:22:48 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389768, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:48.495853 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.478871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:51917 (local address 127.7.43.62:44947)
0810 19:22:48.479086 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:48.479092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:48.480426 (+ 1334us) server_negotiation.cc:407] Connection header received
0810 19:22:48.482412 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:48.482417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:48.482505 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:48.482592 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:48.484245 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.484974 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.486608 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.487042 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.487154 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.490466 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:48.490471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:48.490474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:48.490512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:48.492863 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.493598 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.493602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.493604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.493727 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.494143 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.494146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.494149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.494433 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:48.494567 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:48.495276 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 19:22:48.495517 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:22:48.497087 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.479412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:51917)
0810 19:22:48.480238 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:48.480304 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:48.482136 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:48.482850 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:48.482858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:48.483476 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:48.484067 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:48.484088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.485146 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.485150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:48.486462 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:48.486475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.487261 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.487268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:48.487485 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.488094 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:48.488122 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:48.490262 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:48.492995 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:48.493005 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:48.493024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:48.493438 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:48.493828 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:48.493833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:48.493837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:48.494028 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:48.494562 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:48.494570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:48.494902 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 19:22:48.495149 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 19:22:48.495704 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":328,"threads_started":1}
I20260810 19:22:48.498806 32022 heartbeater.cc:347] Connected to a master server at 127.7.43.62:44947
I20260810 19:22:48.499131 32022 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:48.499992 32022 heartbeater.cc:511] Master 127.7.43.62:44947 requested a full tablet report, sending...
I20260810 19:22:48.501837 31507 ts_manager.cc:194] Registered new tserver with Master: 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2:45655)
I20260810 19:22:48.503031 31507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:51917
W20260810 19:22:48.817449 32028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:48.817781 32028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:48.817904 32028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:48.825109 32028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:48.825219 32028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:48.825346 32028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:22:48.835176 32028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:40499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=35601
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:48.836647 32028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:48.838063 32028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:48.851210 32033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:48.851176 32037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:48.851249 32034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:48.851933 32028 server_base.cc:1034] running on GCE node
I20260810 19:22:48.852512 32028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:48.853449 32028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:48.854699 32028 hybrid_clock.cc:648] HybridClock initialized: now 1786389768854644 us; error 40 us; skew 500 ppm
I20260810 19:22:48.855080 32028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:48 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389768, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:48.858305 32028 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:22:48.859781 32028 webserver.cc:460] Webserver started at http://127.7.43.3:35601/ using document root <none> and password file <none>
I20260810 19:22:48.860510 32028 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:48.860653 32028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:48.866173 32028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:22:48.868865 32049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.870205 32028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:22:48.870366 32028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "3322971d62e14886a8a95f953ba3e77f"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "5284120fd679b9a06cc30b389a7f642c"
server_key_iv: "de18820e95d2940fa471c5bd45148152"
server_key_version: "encryptionkey@0"
I20260810 19:22:48.871430 32028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:48.903585 32028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:48.908041 32028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:48.908325 32028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:48.909317 32028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:48.910799 32028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:48.910900 32028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.911012 32028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:48.911090 32028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:48.941608 32028 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:40499
I20260810 19:22:48.941711 32240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:40499 every 8 connection(s)
I20260810 19:22:48.943136 32028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:22:48.948747 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 32028
Aug 10 19:22:48 dist-test-slave-7lbf krb5kdc[30545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389768, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:22:48.964012 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.949726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.7.43.62:44947)
0810 19:22:48.950202 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:48.950207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:48.950586 (+ 379us) server_negotiation.cc:407] Connection header received
0810 19:22:48.950759 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:48.950763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:48.950835 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:48.950918 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:48.952559 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.953254 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.954903 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.955334 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.955418 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.958309 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:48.958314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:48.958317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:48.958364 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:48.961298 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.961980 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.961986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.961990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.962134 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.962480 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.962485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.962489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.962830 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:48.963075 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:48.963393 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 19:22:48.963607 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"spinlock_wait_cycles":1408,"thread_start_us":248,"threads_started":1}
I20260810 19:22:48.964314 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.945776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:48883 (local address 127.7.43.62:44947)
0810 19:22:48.945957 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:48.945962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:48.947086 (+ 1124us) server_negotiation.cc:407] Connection header received
0810 19:22:48.949036 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:48.949041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:48.949112 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:48.949194 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:48.951281 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.952003 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.953810 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.954219 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.954326 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.958243 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:48.958247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:48.958251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:48.958279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:48.961230 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.961980 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.961984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.961986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.962112 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:48.962572 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:48.962577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:48.962580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:48.962881 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:48.963005 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:48.963734 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 19:22:48.963935 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:22:48.965378 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.946261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:48883)
0810 19:22:48.946932 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:48.946968 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:48.948736 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:48.949443 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:48.949451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:48.950334 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:22:48.951129 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:48.951156 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.952155 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.952160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:48.953687 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:48.953702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.954424 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.954430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:48.954733 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.955398 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:22:48.955430 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:22:48.958034 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:22:48.961366 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:48.961375 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:48.961391 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:48.961862 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:48.962252 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:22:48.962257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:22:48.962259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:22:48.962444 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:22:48.963001 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:22:48.963009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:22:48.963345 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 19:22:48.963574 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 19:22:48.964193 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":265,"threads_started":1}
I20260810 19:22:48.966635 32241 heartbeater.cc:347] Connected to a master server at 127.7.43.62:44947
I20260810 19:22:48.966897 32241 heartbeater.cc:464] Registering TS with master...
I20260810 19:22:48.967551 32241 heartbeater.cc:511] Master 127.7.43.62:44947 requested a full tablet report, sending...
I20260810 19:22:48.968894 31506 ts_manager.cc:194] Registered new tserver with Master: 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499)
I20260810 19:22:48.969843 31506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:48883
I20260810 19:22:48.978605 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:22:48.984894 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:48.979437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.7.43.62:44947)
0810 19:22:48.979579 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:48.979583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:48.980065 (+ 482us) server_negotiation.cc:407] Connection header received
0810 19:22:48.980234 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:48.980237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:48.980366 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:48.980481 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:22:48.980836 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.981583 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.983030 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:48.983432 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:48.983521 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:48.983969 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:22:48.984462 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:22:48.984576 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:48.984613 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 19:22:48.984672 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:22:48.985771 31506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:22:48.992874 31506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:22:49.016111 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.004950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53690)
0810 19:22:49.006162 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.006184 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:49.006294 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:49.007173 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:49.007177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:49.007201 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:49.007672 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.007693 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.009810 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.009815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.012089 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.012102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.014049 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.014056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.014247 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.014302 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 19:22:49.014410 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 19:22:49.014594 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":6,"thread_start_us":416,"threads_started":1}
I20260810 19:22:49.016850 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.004506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.7.43.2:45655)
0810 19:22:49.005089 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.005099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.005928 (+ 829us) server_negotiation.cc:407] Connection header received
0810 19:22:49.006346 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.006354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.006607 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.006781 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:49.007888 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.009601 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.013346 (+ 3745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.014791 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.014889 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.015086 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.016351 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.016536 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":827,"threads_started":1}
I20260810 19:22:49.017001 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.005554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.7.43.1:40111)
0810 19:22:49.006579 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.006586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.006612 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:22:49.006782 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.006789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.006983 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.007871 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:49.008124 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.009677 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.012237 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.013795 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.016202 (+ 2407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.016446 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.016508 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.016660 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":1645,"threads_started":1}
I20260810 19:22:49.016084 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.004338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45655 (local address 127.0.0.1:57984)
0810 19:22:49.005748 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.005823 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:49.006091 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:49.007090 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:49.007097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:49.007128 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:49.007689 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.007698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.009779 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.009784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.013147 (+ 3363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.013163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.015284 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.015291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.015473 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.015505 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:22:49.015587 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:22:49.015708 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":94,"thread_start_us":1404,"threads_started":1}
I20260810 19:22:49.020860 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.006345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:40499 (local address 127.0.0.1:37482)
0810 19:22:49.006915 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.006943 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:49.007106 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:49.010570 (+ 3464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:49.010574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:49.010604 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:49.011089 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.011106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.013005 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.013010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.015032 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.015046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.018017 (+ 2971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.018023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.020223 (+ 2200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.020258 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:22:49.020356 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 19:22:49.020497 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":3,"thread_start_us":998,"threads_started":1}
I20260810 19:22:49.020929 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.006935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.7.43.3:40499)
0810 19:22:49.010019 (+ 3084us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.010027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.010061 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:22:49.010157 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.010165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.010356 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.010553 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:49.011302 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.012846 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.015270 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.017556 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.017665 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.017869 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.020431 (+ 2562us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.020597 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2925,"thread_start_us":2756,"threads_started":1}
I20260810 19:22:49.020538 31913 tablet_service.cc:1467] Processing CreateTablet for tablet f084f4bfed284f149632295c49c31712 (DEFAULT_TABLE table=test-table [id=263e45330b6249258866ed12c588d085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:49.020548 31703 tablet_service.cc:1467] Processing CreateTablet for tablet f084f4bfed284f149632295c49c31712 (DEFAULT_TABLE table=test-table [id=263e45330b6249258866ed12c588d085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:49.022040 31913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f4bfed284f149632295c49c31712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:49.022374 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f4bfed284f149632295c49c31712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:49.024463 32130 tablet_service.cc:1467] Processing CreateTablet for tablet f084f4bfed284f149632295c49c31712 (DEFAULT_TABLE table=test-table [id=263e45330b6249258866ed12c588d085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:22:49.025766 32130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f4bfed284f149632295c49c31712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:22:49.033865 32269 tablet_bootstrap.cc:492] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Bootstrap starting.
I20260810 19:22:49.037415 32271 tablet_bootstrap.cc:492] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Bootstrap starting.
I20260810 19:22:49.037774 32269 tablet_bootstrap.cc:654] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:49.039129 32269 log.cc:824] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:49.040163 32272 tablet_bootstrap.cc:492] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Bootstrap starting.
I20260810 19:22:49.041092 32271 tablet_bootstrap.cc:654] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:49.042299 32271 log.cc:824] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:49.043354 32272 tablet_bootstrap.cc:654] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Neither blocks nor log segments found. Creating new log.
I20260810 19:22:49.044045 32269 tablet_bootstrap.cc:492] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: No bootstrap required, opened a new log
I20260810 19:22:49.044376 32269 ts_tablet_manager.cc:1397] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260810 19:22:49.044525 32272 log.cc:824] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:49.045791 32271 tablet_bootstrap.cc:492] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: No bootstrap required, opened a new log
I20260810 19:22:49.046123 32271 ts_tablet_manager.cc:1397] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260810 19:22:49.048465 32272 tablet_bootstrap.cc:492] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: No bootstrap required, opened a new log
I20260810 19:22:49.048763 32272 ts_tablet_manager.cc:1397] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 19:22:49.050109 31807 heartbeater.cc:503] Master 127.7.43.62:44947 was elected leader, sending a full tablet report...
I20260810 19:22:49.054899 32269 raft_consensus.cc:348] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.055477 32269 raft_consensus.cc:374] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:49.055629 32269 raft_consensus.cc:729] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e87851796246ad873d524198316072, State: Initialized, Role: FOLLOWER
I20260810 19:22:49.056321 32269 consensus_queue.cc:260] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.056509 32271 raft_consensus.cc:348] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.057017 32271 raft_consensus.cc:374] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:49.057174 32271 raft_consensus.cc:729] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e57ecfa88b047ccade60ddde7473261, State: Initialized, Role: FOLLOWER
I20260810 19:22:49.057916 32271 consensus_queue.cc:260] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.058225 32269 ts_tablet_manager.cc:1428] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 19:22:49.058665 32272 raft_consensus.cc:348] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.059140 32272 raft_consensus.cc:374] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:22:49.059149 32022 heartbeater.cc:503] Master 127.7.43.62:44947 was elected leader, sending a full tablet report...
I20260810 19:22:49.059320 32272 raft_consensus.cc:729] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3322971d62e14886a8a95f953ba3e77f, State: Initialized, Role: FOLLOWER
I20260810 19:22:49.059731 32271 ts_tablet_manager.cc:1428] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260810 19:22:49.060103 32272 consensus_queue.cc:260] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.061177 32241 heartbeater.cc:503] Master 127.7.43.62:44947 was elected leader, sending a full tablet report...
I20260810 19:22:49.061898 32272 ts_tablet_manager.cc:1428] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260810 19:22:49.194798 32245 tablet.cc:2369] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:22:49.228263 32023 tablet.cc:2369] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:22:49.266203 31809 tablet.cc:2369] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:22:49.344164 32277 raft_consensus.cc:482] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:22:49.344480 32277 raft_consensus.cc:504] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.346325 32277 leader_election.cc:288] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2:45655), 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499)
I20260810 19:22:49.352612 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.346711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:46035 (local address 127.7.43.2:45655)
0810 19:22:49.346904 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.346909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.347354 (+ 445us) server_negotiation.cc:407] Connection header received
0810 19:22:49.347594 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.347600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.347680 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.347787 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:49.348374 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.349121 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.350949 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.351732 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.351850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.351945 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.352239 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.352356 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:22:49.352723 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.346585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45655 (local address 127.7.43.1:46035)
0810 19:22:49.347260 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.347278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:49.347425 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:49.347848 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:49.347854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:49.347881 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:49.348224 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.348234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.349252 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.349256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.350755 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.350767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.351943 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.351949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.352088 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.352112 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:22:49.352192 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:22:49.352310 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":1,"thread_start_us":356,"threads_started":1}
I20260810 19:22:49.353082 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.347547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:48375 (local address 127.7.43.3:40499)
0810 19:22:49.347734 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.347740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.348106 (+ 366us) server_negotiation.cc:407] Connection header received
0810 19:22:49.348266 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.348272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.348331 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.348404 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:22:49.348929 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.349685 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.351432 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.352210 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.352320 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.352420 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.352711 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.352838 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:22:49.353175 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.347413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:40499 (local address 127.7.43.1:48375)
0810 19:22:49.347990 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.348008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:22:49.348132 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:22:49.348442 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:22:49.348446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:22:49.348469 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:22:49.348779 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.348788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.349814 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.349818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.351287 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:22:49.351299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.352419 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.352426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:22:49.352566 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.352593 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:22:49.352673 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 19:22:49.352792 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":20,"thread_start_us":270,"threads_started":1}
I20260810 19:22:49.353770 31947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f4bfed284f149632295c49c31712" candidate_uuid: "01e87851796246ad873d524198316072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e57ecfa88b047ccade60ddde7473261" is_pre_election: true
I20260810 19:22:49.353955 32166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f4bfed284f149632295c49c31712" candidate_uuid: "01e87851796246ad873d524198316072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3322971d62e14886a8a95f953ba3e77f" is_pre_election: true
I20260810 19:22:49.354240 31947 raft_consensus.cc:2454] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e87851796246ad873d524198316072 in term 0.
I20260810 19:22:49.354410 32166 raft_consensus.cc:2454] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e87851796246ad873d524198316072 in term 0.
I20260810 19:22:49.354839 31628 leader_election.cc:302] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e87851796246ad873d524198316072, 8e57ecfa88b047ccade60ddde7473261; no voters:
I20260810 19:22:49.355187 32277 raft_consensus.cc:2781] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:22:49.355322 32277 raft_consensus.cc:482] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:22:49.355422 32277 raft_consensus.cc:3037] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:49.357717 32277 raft_consensus.cc:504] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.358246 32277 leader_election.cc:288] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [CANDIDATE]: Term 1 election: Requested vote from peers 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2:45655), 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499)
I20260810 19:22:49.358666 31947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f4bfed284f149632295c49c31712" candidate_uuid: "01e87851796246ad873d524198316072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e57ecfa88b047ccade60ddde7473261"
I20260810 19:22:49.358712 32166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f4bfed284f149632295c49c31712" candidate_uuid: "01e87851796246ad873d524198316072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3322971d62e14886a8a95f953ba3e77f"
I20260810 19:22:49.358837 31947 raft_consensus.cc:3037] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:49.358860 32166 raft_consensus.cc:3037] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:22:49.360924 32166 raft_consensus.cc:2454] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e87851796246ad873d524198316072 in term 1.
I20260810 19:22:49.360922 31947 raft_consensus.cc:2454] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e87851796246ad873d524198316072 in term 1.
I20260810 19:22:49.361271 31628 leader_election.cc:302] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e87851796246ad873d524198316072, 8e57ecfa88b047ccade60ddde7473261; no voters:
I20260810 19:22:49.361552 32277 raft_consensus.cc:2781] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:22:49.361958 32277 raft_consensus.cc:686] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 1 LEADER]: Becoming Leader. State: Replica: 01e87851796246ad873d524198316072, State: Running, Role: LEADER
I20260810 19:22:49.362713 32277 consensus_queue.cc:237] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } }
I20260810 19:22:49.367347 31506 catalog_manager.cc:5495] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e87851796246ad873d524198316072 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "01e87851796246ad873d524198316072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e87851796246ad873d524198316072" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260810 19:22:49.469600 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.464762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.7.43.1:40111)
0810 19:22:49.464966 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.464970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.464987 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:22:49.465080 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.465083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.465167 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.465296 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:22:49.465624 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.466314 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.467634 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.468034 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.468121 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.468573 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:22:49.469074 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:22:49.469183 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.469238 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.469304 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:22:49.477969 32166 raft_consensus.cc:1261] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Refusing update from remote peer 01e87851796246ad873d524198316072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:49.478087 31947 raft_consensus.cc:1261] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Refusing update from remote peer 01e87851796246ad873d524198316072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:22:49.478924 32284 consensus_queue.cc:1035] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3322971d62e14886a8a95f953ba3e77f" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:49.479363 32277 consensus_queue.cc:1035] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e57ecfa88b047ccade60ddde7473261" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:22:49.490787 32287 mvcc.cc:205] Tried to move back new op lower bound from 7317052495771664384 to 7317052495313338368. Current Snapshot: MvccSnapshot[applied={T|T < 7317052495771664384}]
I20260810 19:22:49.492797 32288 mvcc.cc:205] Tried to move back new op lower bound from 7317052495771664384 to 7317052495313338368. Current Snapshot: MvccSnapshot[applied={T|T < 7317052495771664384}]
I20260810 19:22:49.496467 32292 mvcc.cc:205] Tried to move back new op lower bound from 7317052495771664384 to 7317052495313338368. Current Snapshot: MvccSnapshot[applied={T|T < 7317052495771664384}]
I20260810 19:22:49.507273 31506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:22:49.507521 31506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:22:49.510972 31506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Sending DeleteTablet for 3 replicas of tablet f084f4bfed284f149632295c49c31712
I20260810 19:22:49.512133 32130 tablet_service.cc:1514] Processing DeleteTablet for tablet f084f4bfed284f149632295c49c31712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:49 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:37482
I20260810 19:22:49.512419 31703 tablet_service.cc:1514] Processing DeleteTablet for tablet f084f4bfed284f149632295c49c31712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:49 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:53690
I20260810 19:22:49.512413 31913 tablet_service.cc:1514] Processing DeleteTablet for tablet f084f4bfed284f149632295c49c31712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:22:49 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:57984
I20260810 19:22:49.513665 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 31560
I20260810 19:22:49.531585 32302 tablet_replica.cc:330] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: stopping tablet replica
I20260810 19:22:49.532229 32301 tablet_replica.cc:330] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: stopping tablet replica
I20260810 19:22:49.532930 32301 raft_consensus.cc:2229] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:22:49.533535 32302 raft_consensus.cc:2229] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:22:49.534301 32301 raft_consensus.cc:2258] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:49.534886 32302 raft_consensus.cc:2258] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:49.535508 32300 tablet_replica.cc:330] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: stopping tablet replica
I20260810 19:22:49.535903 32300 raft_consensus.cc:2229] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:22:49.536365 32300 raft_consensus.cc:2258] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:22:49.539021 32300 ts_tablet_manager.cc:1905] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:49.539028 32301 ts_tablet_manager.cc:1905] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:49.543771 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.514776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.7.43.1:40111)
0810 19:22:49.514964 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.514971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.522169 (+ 7198us) server_negotiation.cc:407] Connection header received
0810 19:22:49.522362 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.522367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.522468 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.522580 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:49.525265 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.526597 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.530655 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.531425 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.531533 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.535578 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:49.535592 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:49.535612 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:49.535652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:49.539450 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:49.540383 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:49.540393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:49.540397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:49.540597 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:49.541056 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:49.541061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:49.541064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:49.541610 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:49.541947 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.543161 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.543397 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:22:49.544741 31798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:49.544968 32302 ts_tablet_manager.cc:1905] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:49.546782 32300 ts_tablet_manager.cc:1918] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:49.547050 32300 log.cc:1196] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/f084f4bfed284f149632295c49c31712
I20260810 19:22:49.547744 32300 ts_tablet_manager.cc:1939] T f084f4bfed284f149632295c49c31712 P 3322971d62e14886a8a95f953ba3e77f: Deleting consensus metadata
I20260810 19:22:49.549806 31480 catalog_manager.cc:4841] TS 3322971d62e14886a8a95f953ba3e77f (127.7.43.3:40499): tablet f084f4bfed284f149632295c49c31712 (table test-table [id=263e45330b6249258866ed12c588d085]) successfully deleted
I20260810 19:22:49.551852 32302 ts_tablet_manager.cc:1918] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:49.551987 32302 log.cc:1196] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/f084f4bfed284f149632295c49c31712
I20260810 19:22:49.552448 32302 ts_tablet_manager.cc:1939] T f084f4bfed284f149632295c49c31712 P 8e57ecfa88b047ccade60ddde7473261: Deleting consensus metadata
I20260810 19:22:49.553931 31480 catalog_manager.cc:4841] TS 8e57ecfa88b047ccade60ddde7473261 (127.7.43.2:45655): tablet f084f4bfed284f149632295c49c31712 (table test-table [id=263e45330b6249258866ed12c588d085]) successfully deleted
I20260810 19:22:49.642042 32301 ts_tablet_manager.cc:1918] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:49.642385 32301 log.cc:1196] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/f084f4bfed284f149632295c49c31712
I20260810 19:22:49.642738 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 31560
W20260810 19:22:49.676247 31485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:40111 (error 108)
I20260810 19:22:49.676466 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 31814
I20260810 19:22:49.678479 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.677763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53722)
0810 19:22:49.678193 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.678299 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":229,"threads_started":1}
W20260810 19:22:49.678679 31485 catalog_manager.cc:4584] TS 01e87851796246ad873d524198316072 (127.7.43.1:40111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f084f4bfed284f149632295c49c31712: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
I20260810 19:22:49.690531 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.677319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.7.43.2:45655)
0810 19:22:49.677497 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.677501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.677651 (+ 150us) server_negotiation.cc:407] Connection header received
0810 19:22:49.677875 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.677881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.677969 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.678067 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:49.679704 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.680461 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.681884 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.682300 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.682421 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.685158 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:49.685174 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:49.685189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:49.685226 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:49.687791 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:49.688454 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:49.688461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:49.688463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:49.688593 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:49.688950 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:49.688954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:49.688956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:49.689359 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:49.689659 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.689979 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.690210 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:22:49.691140 32017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:49.732223 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.731668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53730)
0810 19:22:49.731917 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.732046 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 19:22:49.733289 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.732854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53732)
0810 19:22:49.733032 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.733116 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 19:22:49.778904 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 31814
I20260810 19:22:49.793992 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.793453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53734)
0810 19:22:49.793717 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.793822 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 19:22:49.795063 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.794676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53736)
0810 19:22:49.794829 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.794920 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 19:22:49.804123 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 32028
I20260810 19:22:49.817391 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.804902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.7.43.3:40499)
0810 19:22:49.805109 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 19:22:49.805114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:22:49.805161 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:22:49.805332 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:22:49.805336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:22:49.805445 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:22:49.805579 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:22:49.806903 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.807654 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.808970 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:22:49.809425 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:22:49.809765 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:22:49.812114 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:22:49.812130 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:22:49.812147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:22:49.812185 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:22:49.815138 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:49.815766 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:49.815773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:49.815775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:49.815902 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:22:49.816171 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:22:49.816174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:22:49.816176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:22:49.816496 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:22:49.816701 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 19:22:49.816909 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 19:22:49.817071 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:22:49.818020 32235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:49.876235 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.875786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53738)
0810 19:22:49.875987 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.876080 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 19:22:49.877326 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:49.876876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40111 (local address 127.0.0.1:53750)
0810 19:22:49.877117 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:49.877192 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 19:22:49.904141 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 32028
I20260810 19:22:49.924827 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 31447
I20260810 19:22:49.925968 31552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:22:50.039597 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 31447
I20260810 19:22:50.070729 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44947
--webserver_interface=127.7.43.62
--webserver_port=33381
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:22:50.405105 32317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:50.405421 32317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:50.405586 32317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:50.412951 32317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:22:50.413064 32317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:50.413133 32317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:50.413210 32317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:22:50.413266 32317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:22:50.423282 32317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=33381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:50.424614 32317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:50.426019 32317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:50.437562 32327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:50.437505 32328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:50.437474 32331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:50.438015 32317 server_base.cc:1034] running on GCE node
I20260810 19:22:50.438938 32317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:50.440227 32317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:50.441414 32317 hybrid_clock.cc:648] HybridClock initialized: now 1786389770441355 us; error 50 us; skew 500 ppm
I20260810 19:22:50.441761 32317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:50 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389770, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:50.449306 32317 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:22:50.450810 32317 webserver.cc:460] Webserver started at http://127.7.43.62:33381/ using document root <none> and password file <none>
I20260810 19:22:50.451454 32317 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:50.451565 32317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:50.457062 32317 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 19:22:50.462069 32343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:50.463366 32317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 19:22:50.463516 32317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "a3391a75439e4b4e869d6d9abb592911"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "a6a69892bafe1c97e69aec39fcedcbce"
server_key_iv: "0754bde133f8e7d80cf10c46068567a6"
server_key_version: "encryptionkey@0"
I20260810 19:22:50.464443 32317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:50.485364 32317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:50.489352 32317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:50.489643 32317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:50.506882 32317 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:44947
I20260810 19:22:50.506902 32416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:44947 every 8 connection(s)
I20260810 19:22:50.508476 32317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:22:50.515249 32421 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:22:50.517421 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 32317
I20260810 19:22:50.518239 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:40111
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:44947
--webserver_port=35053
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:50.518554 32421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap starting.
I20260810 19:22:50.540956 32421 log.cc:824] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Log is configured to *not* fsync() on all Append() calls
I20260810 19:22:50.563376 32421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:22:50.564028 32421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911: Bootstrap complete.
I20260810 19:22:50.577284 32421 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:50.578577 32421 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3391a75439e4b4e869d6d9abb592911, State: Initialized, Role: FOLLOWER
I20260810 19:22:50.579311 32421 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:50.579595 32421 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:22:50.579725 32421 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:22:50.579903 32421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 2 FOLLOWER]: Advancing to term 3
I20260810 19:22:50.585575 32421 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:50.586182 32421 leader_election.cc:302] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3391a75439e4b4e869d6d9abb592911; no voters:
I20260810 19:22:50.586849 32421 leader_election.cc:288] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 19:22:50.586984 32429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 19:22:50.588100 32429 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [term 3 LEADER]: Becoming Leader. State: Replica: a3391a75439e4b4e869d6d9abb592911, State: Running, Role: LEADER
I20260810 19:22:50.588644 32429 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } }
I20260810 19:22:50.588989 32421 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:22:50.591149 32436 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3391a75439e4b4e869d6d9abb592911. Latest consensus state: current_term: 3 leader_uuid: "a3391a75439e4b4e869d6d9abb592911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } } }
I20260810 19:22:50.591173 32432 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a3391a75439e4b4e869d6d9abb592911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3391a75439e4b4e869d6d9abb592911" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 44947 } } }
I20260810 19:22:50.591574 32432 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:50.591578 32436 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3391a75439e4b4e869d6d9abb592911 [sys.catalog]: This master's current role is: LEADER
I20260810 19:22:50.602535 32445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:22:50.611146 32445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:22:50.612717 32445 catalog_manager.cc:1259] Loaded cluster ID: 5c73e65d64214116b945157d363ba706
I20260810 19:22:50.612881 32445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 19:22:50.615160 32445 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f52798159ad google::LogMessage::Fail() at ??:0
@ 0x7f527981888a google::LogMessage::SendToLog() at ??:0
@ 0x7f52798154de google::LogMessage::Flush() at ??:0
@ 0x7f5279816a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f528f248512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f528f245976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f528f2cd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f528f2cd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f527c461345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f527c46491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f527c464799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f527c43feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f527f7196db start_thread at ??:0
@ 0x7f52786af61f clone at ??:0
W20260810 19:22:50.940846 32426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:50.941126 32426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:50.941251 32426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:50.948583 32426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:50.948697 32426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:50.948860 32426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:22:50.958835 32426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:40111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=35053
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:50.960140 32426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:50.961489 32426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:50.973899 32499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:50.974072 32501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:50.974409 32503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:50.975368 32426 server_base.cc:1034] running on GCE node
I20260810 19:22:50.975979 32426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:50.976907 32426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:50.978119 32426 hybrid_clock.cc:648] HybridClock initialized: now 1786389770978067 us; error 38 us; skew 500 ppm
I20260810 19:22:50.978435 32426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:50 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389770, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:50.981585 32426 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:22:50.982983 32426 webserver.cc:460] Webserver started at http://127.7.43.1:35053/ using document root <none> and password file <none>
I20260810 19:22:50.983670 32426 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:50.983789 32426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:50.989131 32426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:22:50.991858 32513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:50.993144 32426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:22:50.993304 32426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "01e87851796246ad873d524198316072"
format_stamp: "Formatted at 2026-08-10 19:22:44 on dist-test-slave-7lbf"
server_key: "e0bf29350a02a5cee7d5f8d954e8038a"
server_key_iv: "e217ed454d9801b444655277c36f465f"
server_key_version: "encryptionkey@0"
I20260810 19:22:50.994371 32426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:51.027532 32426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:51.031750 32426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:51.032034 32426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:51.033070 32426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:51.034991 32525 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260810 19:22:51.040661 32525 ts_tablet_manager.cc:1852] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 19:22:51.041741 32525 ts_tablet_manager.cc:1905] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:22:51.049149 32525 ts_tablet_manager.cc:1918] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:22:51.049448 32525 ts_tablet_manager.cc:1939] T f084f4bfed284f149632295c49c31712 P 01e87851796246ad873d524198316072: Deleting consensus metadata
I20260810 19:22:51.049917 32426 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 19:22:51.050053 32426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260810 19:22:51.050176 32426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:51.050254 32426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.082993 32426 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:40111
I20260810 19:22:51.083047 32723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:40111 every 8 connection(s)
I20260810 19:22:51.084340 32426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:22:51.088351 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 32426
I20260810 19:22:51.089179 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:45655
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:44947
--webserver_port=42581
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:51.089708 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.087322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:56995)
0810 19:22:51.088212 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.088452 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":319,"threads_started":1}
W20260810 19:22:51.090425 32727 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:44947 (0 consecutive failures): Network error: Failed to ping master at 127.7.43.62:44947: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
I20260810 19:22:51.091311 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.090929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:53061)
0810 19:22:51.091101 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.091180 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:22:51.092200 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.091855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:60615)
0810 19:22:51.091980 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.092064 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 19:22:51.092413 32727 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:22:51.435163 32731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:51.435469 32731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:51.435626 32731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:51.443114 32731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:51.443246 32731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:51.443428 32731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:22:51.453394 32731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:45655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=42581
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:51.454777 32731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:51.456256 32731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:51.470144 32741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:51.470240 32743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:51.470119 32740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:51.471050 32731 server_base.cc:1034] running on GCE node
I20260810 19:22:51.471663 32731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:51.472612 32731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:51.473805 32731 hybrid_clock.cc:648] HybridClock initialized: now 1786389771473749 us; error 50 us; skew 500 ppm
I20260810 19:22:51.474130 32731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:51 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389771, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:51.477196 32731 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:22:51.478660 32731 webserver.cc:460] Webserver started at http://127.7.43.2:42581/ using document root <none> and password file <none>
I20260810 19:22:51.479367 32731 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:51.479485 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:51.484833 32731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:22:51.487651 32759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.488991 32731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:22:51.489161 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "8e57ecfa88b047ccade60ddde7473261"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "d038b3acfefa32528f7a8592ca5e4c8c"
server_key_iv: "a21d7b98da6516510ffb29f0315131a8"
server_key_version: "encryptionkey@0"
I20260810 19:22:51.490219 32731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:51.505640 32731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:51.509940 32731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:51.510216 32731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:51.511241 32731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:51.512647 32731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:51.512746 32731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.512857 32731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:51.512938 32731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.544337 32731 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45655
I20260810 19:22:51.544409 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45655 every 8 connection(s)
I20260810 19:22:51.545869 32731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:22:51.546846 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 32731
I20260810 19:22:51.547582 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:40499
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:44947
--webserver_port=35601
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:22:51.551347 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.549071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:43273)
0810 19:22:51.549917 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.550179 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":305,"threads_started":1}
W20260810 19:22:51.552073 482 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:44947 (0 consecutive failures): Network error: Failed to ping master at 127.7.43.62:44947: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
I20260810 19:22:51.552954 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.552589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:40527)
0810 19:22:51.552746 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.552831 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:22:51.553807 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.553440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:44907)
0810 19:22:51.553593 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.553682 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:22:51.554086 482 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:22:51.884153 489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:22:51.884449 489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:22:51.884588 489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:22:51.892031 489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:22:51.892175 489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:22:51.892340 489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:22:51.902407 489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:40499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=35601
--tserver_master_addrs=127.7.43.62:44947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:22:51.903739 489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:22:51.905074 489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:22:51.917878 499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:51.918032 502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:22:51.918146 497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:22:51.918985 489 server_base.cc:1034] running on GCE node
I20260810 19:22:51.919591 489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:22:51.920490 489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:22:51.921692 489 hybrid_clock.cc:648] HybridClock initialized: now 1786389771921640 us; error 44 us; skew 500 ppm
I20260810 19:22:51.921986 489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:22:51 dist-test-slave-7lbf krb5kdc[30545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389771, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:22:51.924944 489 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:22:51.926303 489 webserver.cc:460] Webserver started at http://127.7.43.3:35601/ using document root <none> and password file <none>
I20260810 19:22:51.926970 489 fs_manager.cc:359] Metadata directory not provided
I20260810 19:22:51.927086 489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:22:51.932329 489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:22:51.935009 512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.936262 489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:22:51.936412 489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "3322971d62e14886a8a95f953ba3e77f"
format_stamp: "Formatted at 2026-08-10 19:22:45 on dist-test-slave-7lbf"
server_key: "5284120fd679b9a06cc30b389a7f642c"
server_key_iv: "de18820e95d2940fa471c5bd45148152"
server_key_version: "encryptionkey@0"
I20260810 19:22:51.937377 489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:22:51.956239 489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:22:51.960505 489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:22:51.960760 489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:22:51.961792 489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:22:51.963147 489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:22:51.963243 489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.963348 489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:22:51.963426 489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:22:51.993790 489 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:40499
I20260810 19:22:51.993829 711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:40499 every 8 connection(s)
I20260810 19:22:51.995235 489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:22:51.995576 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 489
I20260810 19:22:52.000130 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:51.998348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:37835)
0810 19:22:51.999073 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:51.999264 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":258,"threads_started":1}
W20260810 19:22:52.000731 712 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:44947 (0 consecutive failures): Network error: Failed to ping master at 127.7.43.62:44947: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
I20260810 19:22:52.001497 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:52.001184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:59681)
0810 19:22:52.001322 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:52.001392 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 19:22:52.002301 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:52.001986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:43455)
0810 19:22:52.002119 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:52.002196 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:22:52.002535 712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:22:52.093665 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:52.092882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:36117)
0810 19:22:52.093327 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:52.093408 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260810 19:22:52.555410 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:52.554627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:59903)
0810 19:22:52.555068 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:52.555158 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":213,"threads_started":1}
I20260810 19:22:53.003880 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:53.003101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:52159)
0810 19:22:53.003557 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:53.003643 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":246,"threads_started":1}
I20260810 19:22:53.095230 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:53.094479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:46693)
0810 19:22:53.094950 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:53.095056 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":264,"threads_started":1}
I20260810 19:22:53.556914 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:53.556198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:38113)
0810 19:22:53.556661 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:53.556748 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":234,"threads_started":1}
I20260810 19:22:54.005463 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:54.004691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:49169)
0810 19:22:54.005188 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:54.005289 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
I20260810 19:22:54.096802 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:54.096049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:49377)
0810 19:22:54.096541 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:54.096636 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260810 19:22:54.558460 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:54.557770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:51249)
0810 19:22:54.558223 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:54.558298 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":250,"threads_started":1}
I20260810 19:22:55.007052 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:55.006340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:32935)
0810 19:22:55.006802 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:55.006880 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260810 19:22:55.098356 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:55.097661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:52405)
0810 19:22:55.098101 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:55.098177 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260810 19:22:55.560022 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:55.559308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:54209)
0810 19:22:55.559743 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:55.559821 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260810 19:22:56.008729 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:56.007989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:58607)
0810 19:22:56.008431 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:56.008533 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260810 19:22:56.100036 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:56.099245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:35401)
0810 19:22:56.099726 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:56.099860 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":241,"threads_started":1}
I20260810 19:22:56.561631 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:56.560857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:42149)
0810 19:22:56.561330 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:56.561407 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
I20260810 19:22:57.010301 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:57.009607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:38491)
0810 19:22:57.010060 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:57.010137 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260810 19:22:57.101637 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:57.100892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:34123)
0810 19:22:57.101334 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:57.101413 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260810 19:22:57.563238 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:57.562493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:45697)
0810 19:22:57.562978 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:57.563071 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":243,"threads_started":1}
I20260810 19:22:58.011902 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:58.011200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:50249)
0810 19:22:58.011651 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:58.011741 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260810 19:22:58.103271 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:58.102464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:60091)
0810 19:22:58.102975 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:58.103069 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":235,"threads_started":1}
I20260810 19:22:58.564950 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:58.564101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:43763)
0810 19:22:58.564648 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:58.564729 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":250,"threads_started":1}
I20260810 19:22:59.013687 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:59.012845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:60661)
0810 19:22:59.013365 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:59.013446 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":225,"threads_started":1}
I20260810 19:22:59.104838 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:59.104098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:58761)
0810 19:22:59.104565 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:59.104651 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260810 19:22:59.566723 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:22:59.565967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:46855)
0810 19:22:59.566478 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 19:22:59.566556 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
I20260810 19:23:00.015249 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:00.014527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:44401)
0810 19:23:00.014974 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:00.015064 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":217,"threads_started":1}
I20260810 19:23:00.106446 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:00.105700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:50455)
0810 19:23:00.106186 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:00.106280 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260810 19:23:00.568346 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:00.567596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:57681)
0810 19:23:00.568095 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:00.568172 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":252,"threads_started":1}
I20260810 19:23:01.017010 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:01.016219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:36341)
0810 19:23:01.016716 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:01.016815 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":250,"threads_started":1}
I20260810 19:23:01.108155 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:01.107410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:53003)
0810 19:23:01.107859 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:01.107962 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260810 19:23:01.569989 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:01.569260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:34361)
0810 19:23:01.569742 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:01.569825 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260810 19:23:02.018864 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:02.018059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:36675)
0810 19:23:02.018606 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:02.018701 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
I20260810 19:23:02.109798 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:02.109042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:43529)
0810 19:23:02.109509 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:02.109617 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":259,"threads_started":1}
I20260810 19:23:02.571645 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:02.570893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:55037)
0810 19:23:02.571367 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:02.571462 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
I20260810 19:23:03.020442 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:03.019717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:59959)
0810 19:23:03.020172 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:03.020255 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":231,"threads_started":1}
I20260810 19:23:03.111491 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:03.110725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:42459)
0810 19:23:03.111226 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:03.111308 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
I20260810 19:23:03.573249 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:03.572531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:39313)
0810 19:23:03.573003 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:03.573091 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260810 19:23:04.022118 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:04.021363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:53197)
0810 19:23:04.021868 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:04.021955 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":280,"threads_started":1}
I20260810 19:23:04.113214 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:04.112440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:57943)
0810 19:23:04.112917 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:04.113015 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260810 19:23:04.575047 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:04.574229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:46263)
0810 19:23:04.574745 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:04.574841 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":269,"threads_started":1}
I20260810 19:23:05.023890 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:05.023123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:47019)
0810 19:23:05.023603 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:05.023694 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260810 19:23:05.114912 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:05.114140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:52567)
0810 19:23:05.114620 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:05.114715 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":225,"threads_started":1}
I20260810 19:23:05.576751 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:05.575953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:40677)
0810 19:23:05.576440 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:05.576522 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
I20260810 19:23:06.025569 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:06.024768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:56805)
0810 19:23:06.025273 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:06.025353 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":271,"threads_started":1}
I20260810 19:23:06.116770 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:06.116040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.1:42751)
0810 19:23:06.116512 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:06.116594 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260810 19:23:06.578588 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:06.577906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.2:43711)
0810 19:23:06.578320 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:06.578404 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260810 19:23:06.998037 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 32426
I20260810 19:23:07.021289 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:06.999144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.7.43.1:40111)
0810 19:23:06.999970 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:06.999992 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:07.000043 (+ 51us) server_negotiation.cc:407] Connection header received
0810 19:23:07.001937 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:07.001948 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:07.002396 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:07.002753 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:07.004320 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:07.005617 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:07.006857 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:07.007544 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:07.007717 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:07.014603 (+ 6886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:07.014612 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:07.014629 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:07.014680 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:07.017769 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:07.018540 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:07.018548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:07.018555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:07.018734 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:07.019134 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:07.019138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:07.019140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:07.019726 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:07.020037 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:07.020518 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 19:23:07.020980 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":487,"threads_started":1}
I20260810 19:23:07.022855 32716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:07.027639 795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:07.026768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:44947 (local address 127.7.43.3:40597)
0810 19:23:07.027305 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:07.027438 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
I20260810 19:23:07.109241 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 32426
I20260810 19:23:07.135046 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 32731
I20260810 19:23:07.152205 796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:07.135995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.7.43.2:45655)
0810 19:23:07.136501 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:07.136515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:07.136554 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:23:07.138087 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:07.138098 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:07.138524 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:07.138854 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:07.140306 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:07.141648 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:07.142878 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:07.143552 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:07.143728 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:07.146253 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:07.146262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:07.146277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:07.146321 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:07.148940 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:07.149626 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:07.149635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:07.149642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:07.149813 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:07.150234 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:07.150238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:07.150240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:07.150768 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:07.151072 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:07.151492 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 19:23:07.151917 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
I20260810 19:23:07.153625 474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:07.239593 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 32731
I20260810 19:23:07.263972 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 489
I20260810 19:23:07.280766 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:07.264939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.7.43.3:40499)
0810 19:23:07.265588 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:07.265604 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:07.265646 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:23:07.267244 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:07.267253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:07.267704 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:07.268000 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:07.269351 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:07.270728 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:07.271931 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:07.272573 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:07.272702 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:07.275114 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:07.275122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:07.275137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:07.275176 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:07.277974 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:07.278578 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:07.278584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:07.278590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:07.278753 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:07.279051 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:07.279054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:07.279056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:07.279561 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:07.279775 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:07.280054 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 19:23:07.280457 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
I20260810 19:23:07.282155 706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:07.363304 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 489
2026-08-10T19:23:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25134 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:23:07 dist-test-slave-7lbf krb5kdc[807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:23:07 dist-test-slave-7lbf krb5kdc[807](info): set up 2 sockets
Aug 10 19:23:07 dist-test-slave-7lbf krb5kdc[807](info): commencing operation
krb5kdc: starting...
W20260810 19:23:08.619827 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:23:08 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:23:08.927110 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36245
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:09.269824 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:09.270136 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:09.270265 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:09.277724 830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:23:09.277827 830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:23:09.277889 830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:23:09.277968 830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:23:09.278038 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:09.278131 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:09.278191 830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:23:09.278266 830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:23:09.288324 830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:09.289667 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:09.291105 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:09.302906 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:09.302991 839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:09.302999 842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:09.303683 830 server_base.cc:1034] running on GCE node
I20260810 19:23:09.304462 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:09.305909 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:09.307137 830 hybrid_clock.cc:648] HybridClock initialized: now 1786389789307078 us; error 46 us; skew 500 ppm
I20260810 19:23:09.307459 830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:09 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:09.310659 830 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:23:09.312214 830 webserver.cc:460] Webserver started at http://127.7.43.62:41023/ using document root <none> and password file <none>
I20260810 19:23:09.312908 830 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:09.313023 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:09.313328 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:23:09.316432 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "07730a3a82f54f2b94153c1184a50747"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "c13cce55b31b45f6b2da569b1e6dbd02"
server_key_iv: "4b029c7cfa66de8a6a6ad5ce144fa9fe"
server_key_version: "encryptionkey@0"
I20260810 19:23:09.317199 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "07730a3a82f54f2b94153c1184a50747"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "c13cce55b31b45f6b2da569b1e6dbd02"
server_key_iv: "4b029c7cfa66de8a6a6ad5ce144fa9fe"
server_key_version: "encryptionkey@0"
I20260810 19:23:09.322539 830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:23:09.326051 860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:09.327363 830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:23:09.327534 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "07730a3a82f54f2b94153c1184a50747"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "c13cce55b31b45f6b2da569b1e6dbd02"
server_key_iv: "4b029c7cfa66de8a6a6ad5ce144fa9fe"
server_key_version: "encryptionkey@0"
I20260810 19:23:09.327741 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:09.348012 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:09.352325 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:09.352581 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:09.370702 830 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:36245
I20260810 19:23:09.370714 938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:36245 every 8 connection(s)
I20260810 19:23:09.372156 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:23:09.375595 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 830
I20260810 19:23:09.375825 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:23:09.376327 942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:09.376695 7340 external_mini_cluster.cc:1428] Setting key eb16e47f99316fdc98f07cb134479728
I20260810 19:23:09.391238 942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap starting.
Aug 10 19:23:09 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:09.395249 942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:09.396668 942 log.cc:824] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:09.400197 942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: No bootstrap required, opened a new log
I20260810 19:23:09.401417 945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:09.378793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.7.43.62:36245)
0810 19:23:09.379664 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:09.379676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:09.379718 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:23:09.381346 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:09.381370 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:09.381838 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:09.382179 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:09.384050 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:09.385318 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:09.387199 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:09.387933 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:09.388071 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:09.392863 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:09.392875 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:09.392894 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:09.392947 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:09.396822 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:09.397620 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:09.397627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:09.397633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:09.397798 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:09.398234 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:09.398239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:09.398241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:09.398927 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:09.399215 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:09.399557 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 19:23:09.400035 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":393,"threads_started":1}
I20260810 19:23:09.408516 942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:09.408857 942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:09.408931 942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07730a3a82f54f2b94153c1184a50747, State: Initialized, Role: FOLLOWER
I20260810 19:23:09.409435 942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:09.409688 942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:23:09.409792 942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:23:09.409922 942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:09.412299 942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:09.412783 942 leader_election.cc:302] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07730a3a82f54f2b94153c1184a50747; no voters:
I20260810 19:23:09.413362 942 leader_election.cc:288] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:23:09.413501 952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:23:09.414145 952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 LEADER]: Becoming Leader. State: Replica: 07730a3a82f54f2b94153c1184a50747, State: Running, Role: LEADER
I20260810 19:23:09.414767 952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:09.415640 942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:23:09.417598 957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07730a3a82f54f2b94153c1184a50747. Latest consensus state: current_term: 1 leader_uuid: "07730a3a82f54f2b94153c1184a50747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } } }
I20260810 19:23:09.417687 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07730a3a82f54f2b94153c1184a50747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } } }
I20260810 19:23:09.418040 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: This master's current role is: LEADER
I20260810 19:23:09.418017 957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: This master's current role is: LEADER
I20260810 19:23:09.419996 965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:23:09.425629 965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:23:09.434749 965 catalog_manager.cc:1347] Generated new cluster ID: 22755aa2e51d45de8e70bbbe23a5da3c
I20260810 19:23:09.434890 965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:23:09.449965 965 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:23:09.451373 965 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:23:09.460470 965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Generated new TSK 0
I20260810 19:23:09.461299 965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:23:09.541100 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36245
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:09.896088 995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:09.896404 995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:09.896515 995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:09.904035 995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:09.904158 995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:09.904284 995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:23:09.914196 995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:09.915520 995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:09.916893 995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:09.929880 1003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:09.929827 1006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:09.929909 1008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:09.930567 995 server_base.cc:1034] running on GCE node
I20260810 19:23:09.931246 995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:09.932230 995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:09.933445 995 hybrid_clock.cc:648] HybridClock initialized: now 1786389789933380 us; error 48 us; skew 500 ppm
I20260810 19:23:09.933789 995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:09 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:09.936882 995 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:23:09.938277 995 webserver.cc:460] Webserver started at http://127.7.43.1:44345/ using document root <none> and password file <none>
I20260810 19:23:09.938937 995 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:09.939036 995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:09.939302 995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:23:09.942250 995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "8dc9fbe8df0847d985170952e5932353"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "82e5adcff0c4b886871813e9007b3188"
server_key_iv: "baf3265c3f53899ce9b74f881ae7d61b"
server_key_version: "encryptionkey@0"
I20260810 19:23:09.942986 995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "8dc9fbe8df0847d985170952e5932353"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "82e5adcff0c4b886871813e9007b3188"
server_key_iv: "baf3265c3f53899ce9b74f881ae7d61b"
server_key_version: "encryptionkey@0"
I20260810 19:23:09.948222 995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:23:09.951781 1016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:09.953102 995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:23:09.953236 995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "8dc9fbe8df0847d985170952e5932353"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "82e5adcff0c4b886871813e9007b3188"
server_key_iv: "baf3265c3f53899ce9b74f881ae7d61b"
server_key_version: "encryptionkey@0"
I20260810 19:23:09.953406 995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:09.971617 995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:09.975687 995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:09.975950 995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:09.976958 995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:09.978315 995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:09.978387 995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:09.978492 995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:09.978534 995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:10.008713 995 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:41089
I20260810 19:23:10.008759 1201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:41089 every 8 connection(s)
I20260810 19:23:10.010033 995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:23:10.016232 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 995
I20260810 19:23:10.016466 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:23:10.017305 7340 external_mini_cluster.cc:1428] Setting key a8cf87e5daee92acad3239c32a511ba2
Aug 10 19:23:10 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389789, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:10.031354 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:10.012601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:40377 (local address 127.7.43.62:36245)
0810 19:23:10.013048 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:10.013053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:10.014052 (+ 999us) server_negotiation.cc:407] Connection header received
0810 19:23:10.015978 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:10.015982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:10.016060 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:10.016155 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:10.018218 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.019232 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.021309 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.021796 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.021894 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:10.025869 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:10.025874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:10.025878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:10.025908 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:10.028346 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:10.029047 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:10.029051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:10.029053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:10.029174 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:10.029603 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:10.029607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:10.029609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:10.029880 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:10.030026 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:10.030737 (+ 711us) server_negotiation.cc:299] Negotiation successful
0810 19:23:10.031005 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":214,"threads_started":1}
I20260810 19:23:10.032320 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:10.013134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:40377)
0810 19:23:10.013886 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:10.013935 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:10.015661 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:10.016437 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:10.016449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:10.017191 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:10.018044 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:10.018066 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.019373 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.019377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:10.021178 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:10.021192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.022019 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.022026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:10.022298 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:10.023003 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:10.023038 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:10.025650 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:10.028486 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:10.028495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:10.028513 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:10.028919 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:10.029277 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:10.029282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:10.029286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:10.029482 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:10.030005 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:10.030013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:10.030345 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 19:23:10.030612 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 19:23:10.031158 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
I20260810 19:23:10.033895 1202 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36245
I20260810 19:23:10.034220 1202 heartbeater.cc:464] Registering TS with master...
I20260810 19:23:10.035082 1202 heartbeater.cc:511] Master 127.7.43.62:36245 requested a full tablet report, sending...
I20260810 19:23:10.037271 885 ts_manager.cc:194] Registered new tserver with Master: 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089)
I20260810 19:23:10.038757 885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:40377
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:23:10.084022 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36245
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:10.421463 1212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:10.421839 1212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:10.421962 1212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:10.429337 1212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:10.429456 1212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:10.429623 1212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:23:10.439674 1212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:10.441021 1212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:10.442392 1212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:10.455407 1218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:10.455440 1219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:10.455478 1221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:10.456130 1212 server_base.cc:1034] running on GCE node
I20260810 19:23:10.456862 1212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:10.457934 1212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:10.459138 1212 hybrid_clock.cc:648] HybridClock initialized: now 1786389790459094 us; error 32 us; skew 500 ppm
I20260810 19:23:10.459448 1212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:10 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:10.462646 1212 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:23:10.464181 1212 webserver.cc:460] Webserver started at http://127.7.43.2:36349/ using document root <none> and password file <none>
I20260810 19:23:10.464845 1212 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:10.464947 1212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:10.465210 1212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:23:10.468223 1212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "16bd5000dd1a488997d78ba43b14ae16"
format_stamp: "Formatted at 2026-08-10 19:23:10 on dist-test-slave-7lbf"
server_key: "0a9f6290caaa8ddd5db2f8116b8a5cd0"
server_key_iv: "c0658f06854dfc2dc7a2577f61e5161b"
server_key_version: "encryptionkey@0"
I20260810 19:23:10.469013 1212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "16bd5000dd1a488997d78ba43b14ae16"
format_stamp: "Formatted at 2026-08-10 19:23:10 on dist-test-slave-7lbf"
server_key: "0a9f6290caaa8ddd5db2f8116b8a5cd0"
server_key_iv: "c0658f06854dfc2dc7a2577f61e5161b"
server_key_version: "encryptionkey@0"
I20260810 19:23:10.474530 1212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 19:23:10.478411 1231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:10.479846 1212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:23:10.480012 1212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "16bd5000dd1a488997d78ba43b14ae16"
format_stamp: "Formatted at 2026-08-10 19:23:10 on dist-test-slave-7lbf"
server_key: "0a9f6290caaa8ddd5db2f8116b8a5cd0"
server_key_iv: "c0658f06854dfc2dc7a2577f61e5161b"
server_key_version: "encryptionkey@0"
I20260810 19:23:10.480206 1212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:10.501667 1212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:10.506178 1212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:10.506455 1212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:10.507462 1212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:10.508821 1212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:10.508898 1212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:10.509018 1212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:10.509068 1212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:10.540302 1212 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:35729
I20260810 19:23:10.540351 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:35729 every 8 connection(s)
I20260810 19:23:10.541636 1212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:23:10.551916 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 1212
I20260810 19:23:10.552120 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:23:10.552626 7340 external_mini_cluster.cc:1428] Setting key 20b548bae080a7f77798d23b41a076fa
Aug 10 19:23:10 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389790, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:10.561409 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:10.544240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:58567 (local address 127.7.43.62:36245)
0810 19:23:10.544673 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:10.544677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:10.545583 (+ 906us) server_negotiation.cc:407] Connection header received
0810 19:23:10.547427 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:10.547443 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:10.547534 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:10.547630 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:10.549191 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.549965 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.551551 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.551994 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.552108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:10.555845 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:10.555850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:10.555854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:10.555885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:10.558438 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:10.559123 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:10.559127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:10.559129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:10.559259 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:10.559674 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:10.559678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:10.559680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:10.559935 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:10.560076 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:10.560786 (+ 710us) server_negotiation.cc:299] Negotiation successful
0810 19:23:10.561023 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":235,"threads_started":1}
I20260810 19:23:10.562448 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:10.544715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:58567)
0810 19:23:10.545409 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:10.545448 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:10.547157 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:10.547889 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:10.547897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:10.548421 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:10.549019 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:10.549038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.550115 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.550119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:10.551386 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:10.551399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:10.552223 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:10.552230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:10.552458 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:10.553127 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:10.553159 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:10.555622 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:10.558571 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:10.558581 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:10.558599 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:10.559009 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:10.559362 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:10.559367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:10.559371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:10.559568 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:10.560060 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:10.560068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:10.560419 (+ 351us) client_negotiation.cc:769] Sending connection context
0810 19:23:10.560682 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 19:23:10.561308 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":254,"threads_started":1}
I20260810 19:23:10.563898 1418 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36245
I20260810 19:23:10.564242 1418 heartbeater.cc:464] Registering TS with master...
I20260810 19:23:10.565094 1418 heartbeater.cc:511] Master 127.7.43.62:36245 requested a full tablet report, sending...
I20260810 19:23:10.566637 885 ts_manager.cc:194] Registered new tserver with Master: 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729)
I20260810 19:23:10.567430 885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:58567
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:23:10.620842 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36245
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:10.970469 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:10.970767 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:10.970870 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:10.978180 1427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:10.978293 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:10.978418 1427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:23:10.988552 1427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:10.989976 1427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:10.991319 1427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:11.004609 1435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:11.004536 1437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:11.004586 1439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:11.005460 1427 server_base.cc:1034] running on GCE node
I20260810 19:23:11.006124 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:11.007081 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:11.008322 1427 hybrid_clock.cc:648] HybridClock initialized: now 1786389791008256 us; error 59 us; skew 500 ppm
I20260810 19:23:11.008697 1427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:11 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:11.012967 1427 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:23:11.014827 1427 webserver.cc:460] Webserver started at http://127.7.43.3:37587/ using document root <none> and password file <none>
I20260810 19:23:11.015507 1427 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:11.015618 1427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:11.015942 1427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:23:11.018936 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "7a1386f40f734df89b4135cf7a34b356"
format_stamp: "Formatted at 2026-08-10 19:23:11 on dist-test-slave-7lbf"
server_key: "b6c009da7f2d85691e28612c17df871d"
server_key_iv: "50fa4f098fe46e7a088d76b4097452d8"
server_key_version: "encryptionkey@0"
I20260810 19:23:11.019654 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "7a1386f40f734df89b4135cf7a34b356"
format_stamp: "Formatted at 2026-08-10 19:23:11 on dist-test-slave-7lbf"
server_key: "b6c009da7f2d85691e28612c17df871d"
server_key_iv: "50fa4f098fe46e7a088d76b4097452d8"
server_key_version: "encryptionkey@0"
I20260810 19:23:11.024884 1427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:23:11.028424 1455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:11.029788 1427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:23:11.029932 1427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "7a1386f40f734df89b4135cf7a34b356"
format_stamp: "Formatted at 2026-08-10 19:23:11 on dist-test-slave-7lbf"
server_key: "b6c009da7f2d85691e28612c17df871d"
server_key_iv: "50fa4f098fe46e7a088d76b4097452d8"
server_key_version: "encryptionkey@0"
I20260810 19:23:11.030117 1427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:11.042285 1202 heartbeater.cc:503] Master 127.7.43.62:36245 was elected leader, sending a full tablet report...
I20260810 19:23:11.071206 1427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:11.075265 1427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:11.075569 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:11.076576 1427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:11.078037 1427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:11.078118 1427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:11.078246 1427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:11.078310 1427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:11.109288 1427 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:35497
I20260810 19:23:11.109335 1643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:35497 every 8 connection(s)
I20260810 19:23:11.110627 1427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:23:11.111966 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 1427
I20260810 19:23:11.112154 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:23:11.112620 7340 external_mini_cluster.cc:1428] Setting key 9cea23f05507af4334024b063df5ad37
Aug 10 19:23:11 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389791, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:11.132030 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.113262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:47331 (local address 127.7.43.62:36245)
0810 19:23:11.113944 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.113950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.114877 (+ 927us) server_negotiation.cc:407] Connection header received
0810 19:23:11.117279 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.117284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.117393 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.117494 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:11.119358 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.120165 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.121929 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.122331 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.122465 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.125947 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:11.125952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:11.125955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:11.125992 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:11.128940 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.129691 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.129696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.129698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.129819 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.130306 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.130309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.130311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.130587 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:11.130697 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.131501 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.131701 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":329,"threads_started":1}
I20260810 19:23:11.133046 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.113918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:47331)
0810 19:23:11.114696 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:11.114755 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:11.116964 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:11.117816 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:11.117828 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:11.118592 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:11.119193 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.119211 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.120328 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.120334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.121784 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.121798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.122567 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.122575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.122839 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.123455 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:11.123477 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:11.125706 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:11.129065 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:11.129073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:11.129087 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:11.129597 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:11.129984 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:11.129989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:11.129991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:11.130217 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:11.130700 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:11.130707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:11.131084 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 19:23:11.131373 (+ 289us) client_negotiation.cc:240] Negotiation successful
0810 19:23:11.131878 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":289,"threads_started":1}
I20260810 19:23:11.134508 1645 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36245
I20260810 19:23:11.134809 1645 heartbeater.cc:464] Registering TS with master...
I20260810 19:23:11.135528 1645 heartbeater.cc:511] Master 127.7.43.62:36245 requested a full tablet report, sending...
I20260810 19:23:11.136902 885 ts_manager.cc:194] Registered new tserver with Master: 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
I20260810 19:23:11.137725 885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:47331
I20260810 19:23:11.141067 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:23:11.158350 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.145618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.7.43.62:36245)
0810 19:23:11.145786 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.145793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.146220 (+ 427us) server_negotiation.cc:407] Connection header received
0810 19:23:11.146399 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.146403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.146467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.146547 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:11.147966 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.148703 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.150201 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.150646 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.150758 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.153188 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:11.153193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:11.153197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:11.153230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:11.155748 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.156413 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.156417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.156420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.156592 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.156889 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.156892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.156894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.157242 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:11.157475 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.157905 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.158089 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:23:11.162176 885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:23:11.169107 885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:23:11.190979 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.182281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.7.43.3:35497)
0810 19:23:11.183731 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.183740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.183769 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:23:11.183862 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.183869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.184094 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.184933 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:11.185187 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.186443 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.188798 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.189866 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.189991 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.190217 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.190459 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.190630 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":1090,"threads_started":1}
I20260810 19:23:11.191089 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.181851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:35497 (local address 127.0.0.1:50776)
0810 19:23:11.182411 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:11.182428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:11.182536 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:11.184293 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:11.184298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:11.184320 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:11.184734 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.184748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.186592 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.186597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.188639 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.188654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.190042 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.190049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.190243 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.190296 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 19:23:11.190410 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 19:23:11.190613 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":54,"thread_start_us":316,"threads_started":1}
I20260810 19:23:11.191150 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.180583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35729 (local address 127.0.0.1:56764)
0810 19:23:11.181426 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:11.181480 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:11.181664 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:11.182784 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:11.182793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:11.182831 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:11.183263 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.183275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.185919 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.185924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.188550 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.188569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.190312 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.190319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.190478 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.190522 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 19:23:11.190594 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 19:23:11.190718 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":68,"thread_start_us":482,"threads_started":1}
I20260810 19:23:11.191781 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.182471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:38880)
0810 19:23:11.182994 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:11.183013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:11.183123 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:11.184686 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:11.184691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:11.184720 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:11.185063 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.185074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.186733 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.186738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.188952 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.188966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.191028 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.191034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.191206 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.191236 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 19:23:11.191320 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 19:23:11.191434 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":42,"thread_start_us":625,"threads_started":1}
I20260810 19:23:11.192459 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.182956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.7.43.1:41089)
0810 19:23:11.184058 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.184089 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.184117 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:23:11.184208 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.184215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.184459 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.185244 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:11.185452 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.186592 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.189107 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.190499 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.190630 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.190857 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.191988 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.192163 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":757,"threads_started":1}
I20260810 19:23:11.192781 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.180726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.7.43.2:35729)
0810 19:23:11.182111 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.182120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.182149 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:23:11.182247 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.182254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.182514 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.183467 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:11.184208 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.185785 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.188731 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.190083 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.191350 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.192295 (+ 945us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.192359 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.192520 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":1093,"threads_started":1}
I20260810 19:23:11.195192 1546 tablet_service.cc:1467] Processing CreateTablet for tablet e8ea6306a7c340f1b36786584552ed6b (DEFAULT_TABLE table=test-table [id=192eb9a7718742a6bc47fbd06abf0d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:23:11.196230 1306 tablet_service.cc:1467] Processing CreateTablet for tablet e8ea6306a7c340f1b36786584552ed6b (DEFAULT_TABLE table=test-table [id=192eb9a7718742a6bc47fbd06abf0d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:23:11.196287 1097 tablet_service.cc:1467] Processing CreateTablet for tablet e8ea6306a7c340f1b36786584552ed6b (DEFAULT_TABLE table=test-table [id=192eb9a7718742a6bc47fbd06abf0d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:23:11.196825 1546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ea6306a7c340f1b36786584552ed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:11.197683 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ea6306a7c340f1b36786584552ed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:11.197714 1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8ea6306a7c340f1b36786584552ed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:11.208496 1677 tablet_bootstrap.cc:492] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Bootstrap starting.
I20260810 19:23:11.211423 1680 tablet_bootstrap.cc:492] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Bootstrap starting.
I20260810 19:23:11.212215 1678 tablet_bootstrap.cc:492] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Bootstrap starting.
I20260810 19:23:11.213336 1677 tablet_bootstrap.cc:654] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:11.214720 1677 log.cc:824] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:11.215735 1678 tablet_bootstrap.cc:654] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:11.216356 1680 tablet_bootstrap.cc:654] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:11.216938 1678 log.cc:824] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:11.217458 1680 log.cc:824] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:11.218257 1677 tablet_bootstrap.cc:492] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: No bootstrap required, opened a new log
I20260810 19:23:11.218560 1677 ts_tablet_manager.cc:1397] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 19:23:11.220450 1678 tablet_bootstrap.cc:492] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: No bootstrap required, opened a new log
I20260810 19:23:11.220733 1678 ts_tablet_manager.cc:1397] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 19:23:11.227982 1680 tablet_bootstrap.cc:492] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: No bootstrap required, opened a new log
I20260810 19:23:11.228262 1680 ts_tablet_manager.cc:1397] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260810 19:23:11.229126 1677 raft_consensus.cc:348] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.229646 1677 raft_consensus.cc:374] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:11.229748 1677 raft_consensus.cc:729] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc9fbe8df0847d985170952e5932353, State: Initialized, Role: FOLLOWER
I20260810 19:23:11.230425 1677 consensus_queue.cc:260] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.231298 1678 raft_consensus.cc:348] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.231729 1678 raft_consensus.cc:374] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:11.231827 1678 raft_consensus.cc:729] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bd5000dd1a488997d78ba43b14ae16, State: Initialized, Role: FOLLOWER
I20260810 19:23:11.232332 1677 ts_tablet_manager.cc:1428] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 19:23:11.232547 1678 consensus_queue.cc:260] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.237733 1680 raft_consensus.cc:348] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.238087 1680 raft_consensus.cc:374] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:11.238181 1680 raft_consensus.cc:729] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1386f40f734df89b4135cf7a34b356, State: Initialized, Role: FOLLOWER
I20260810 19:23:11.238771 1680 consensus_queue.cc:260] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.239645 1418 heartbeater.cc:503] Master 127.7.43.62:36245 was elected leader, sending a full tablet report...
I20260810 19:23:11.240028 1645 heartbeater.cc:503] Master 127.7.43.62:36245 was elected leader, sending a full tablet report...
I20260810 19:23:11.240711 1678 ts_tablet_manager.cc:1428] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260810 19:23:11.240726 1680 ts_tablet_manager.cc:1428] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260810 19:23:11.262388 1203 tablet.cc:2369] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:23:11.293598 1419 tablet.cc:2369] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:23:11.307247 1691 raft_consensus.cc:482] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:23:11.307549 1691 raft_consensus.cc:504] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.309454 1691 leader_election.cc:288] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089)
I20260810 19:23:11.315886 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.309807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35729 (local address 127.7.43.3:46093)
0810 19:23:11.310139 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:11.310159 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:11.310289 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:11.310768 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:11.310772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:11.310803 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:11.310998 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.311010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.312237 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.312240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.314015 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.314031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.315296 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.315300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.315437 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.315461 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:23:11.315540 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:23:11.315677 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 19:23:11.315961 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.309891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:46093 (local address 127.7.43.2:35729)
0810 19:23:11.310143 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.310149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.310242 (+ 93us) server_negotiation.cc:407] Connection header received
0810 19:23:11.310421 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.310426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.310485 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.310582 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:11.311180 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.312097 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.314161 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.315146 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.315280 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.315377 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.315580 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.315711 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 19:23:11.316962 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.310570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:33481 (local address 127.7.43.1:41089)
0810 19:23:11.310771 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.310777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.311103 (+ 326us) server_negotiation.cc:407] Connection header received
0810 19:23:11.311283 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.311288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.311364 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.311463 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:11.311900 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.312685 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.314747 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.315962 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.316113 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.316208 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.316597 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.316716 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:23:11.317013 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.310248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.7.43.3:33481)
0810 19:23:11.310990 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:11.311037 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:11.311147 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:11.311472 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:11.311475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:11.311493 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:11.311763 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.311770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.312834 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.312839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.314589 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:11.314602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.316317 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.316321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:11.316445 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.316469 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:23:11.316545 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 19:23:11.316663 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":1,"thread_start_us":409,"threads_started":1}
I20260810 19:23:11.316946 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ea6306a7c340f1b36786584552ed6b" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd5000dd1a488997d78ba43b14ae16" is_pre_election: true
I20260810 19:23:11.317507 1344 raft_consensus.cc:2454] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 0.
I20260810 19:23:11.318009 1126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ea6306a7c340f1b36786584552ed6b" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc9fbe8df0847d985170952e5932353" is_pre_election: true
I20260810 19:23:11.318226 1468 leader_election.cc:302] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356; no voters:
I20260810 19:23:11.318583 1126 raft_consensus.cc:2454] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 0.
I20260810 19:23:11.318641 1691 raft_consensus.cc:2781] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:23:11.318770 1691 raft_consensus.cc:482] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:23:11.318859 1691 raft_consensus.cc:3037] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:11.321180 1691 raft_consensus.cc:504] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.321820 1691 leader_election.cc:288] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 election: Requested vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089)
I20260810 19:23:11.322186 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ea6306a7c340f1b36786584552ed6b" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd5000dd1a488997d78ba43b14ae16"
I20260810 19:23:11.322264 1126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8ea6306a7c340f1b36786584552ed6b" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc9fbe8df0847d985170952e5932353"
I20260810 19:23:11.322368 1344 raft_consensus.cc:3037] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:11.322449 1126 raft_consensus.cc:3037] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:11.324561 1344 raft_consensus.cc:2454] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 1.
I20260810 19:23:11.324664 1126 raft_consensus.cc:2454] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 1.
I20260810 19:23:11.324973 1468 leader_election.cc:302] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356; no voters:
I20260810 19:23:11.325246 1691 raft_consensus.cc:2781] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:23:11.325702 1691 raft_consensus.cc:686] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 1 LEADER]: Becoming Leader. State: Replica: 7a1386f40f734df89b4135cf7a34b356, State: Running, Role: LEADER
I20260810 19:23:11.326161 1691 consensus_queue.cc:237] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } }
I20260810 19:23:11.330626 885 catalog_manager.cc:5495] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "7a1386f40f734df89b4135cf7a34b356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20260810 19:23:11.351089 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.346097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.7.43.3:35497)
0810 19:23:11.346299 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.346305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.346378 (+ 73us) server_negotiation.cc:407] Connection header received
0810 19:23:11.346551 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.346555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.346642 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.346766 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:23:11.347047 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.347742 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.349087 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.349467 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.349614 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.350085 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:23:11.350614 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:23:11.350719 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.350774 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.350839 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:23:11.359110 1126 raft_consensus.cc:1261] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Refusing update from remote peer 7a1386f40f734df89b4135cf7a34b356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:23:11.359148 1344 raft_consensus.cc:1261] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Refusing update from remote peer 7a1386f40f734df89b4135cf7a34b356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:23:11.359997 1691 consensus_queue.cc:1035] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:23:11.360359 1693 consensus_queue.cc:1035] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:23:11.362658 1646 tablet.cc:2369] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:23:11.371603 1697 mvcc.cc:205] Tried to move back new op lower bound from 7317052585396879360 to 7317052585276149760. Current Snapshot: MvccSnapshot[applied={T|T < 7317052585396879360}]
I20260810 19:23:11.396811 884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:23:11.397176 884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:23:11.401100 884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Sending DeleteTablet for 3 replicas of tablet e8ea6306a7c340f1b36786584552ed6b
I20260810 19:23:11.402215 1306 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ea6306a7c340f1b36786584552ed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:23:11 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:56764
I20260810 19:23:11.402395 1097 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ea6306a7c340f1b36786584552ed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:23:11 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:38880
I20260810 19:23:11.402555 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 995
I20260810 19:23:11.403985 1546 tablet_service.cc:1514] Processing DeleteTablet for tablet e8ea6306a7c340f1b36786584552ed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:23:11 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:50776
I20260810 19:23:11.412777 1714 tablet_replica.cc:330] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: stopping tablet replica
I20260810 19:23:11.414705 1716 tablet_replica.cc:330] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: stopping tablet replica
I20260810 19:23:11.415158 1716 raft_consensus.cc:2229] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:23:11.413395 1714 raft_consensus.cc:2229] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:23:11.416327 1714 raft_consensus.cc:2258] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:23:11.419157 1714 ts_tablet_manager.cc:1905] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:23:11.419625 1716 raft_consensus.cc:2258] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:23:11.422117 1716 ts_tablet_manager.cc:1905] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:23:11.423543 1713 tablet_replica.cc:330] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: stopping tablet replica
I20260810 19:23:11.423987 1713 raft_consensus.cc:2229] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:23:11.424532 1713 raft_consensus.cc:2258] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:23:11.427709 1713 ts_tablet_manager.cc:1905] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:23:11.428288 1714 ts_tablet_manager.cc:1918] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:23:11.428478 1714 log.cc:1196] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/e8ea6306a7c340f1b36786584552ed6b
I20260810 19:23:11.429041 1714 ts_tablet_manager.cc:1939] T e8ea6306a7c340f1b36786584552ed6b P 8dc9fbe8df0847d985170952e5932353: Deleting consensus metadata
Aug 10 19:23:11 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:23:11.431147 865 catalog_manager.cc:4841] TS 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089): tablet e8ea6306a7c340f1b36786584552ed6b (table test-table [id=192eb9a7718742a6bc47fbd06abf0d8c]) successfully deleted
I20260810 19:23:11.435743 1713 ts_tablet_manager.cc:1918] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:23:11.435953 1713 log.cc:1196] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/e8ea6306a7c340f1b36786584552ed6b
I20260810 19:23:11.436479 1713 ts_tablet_manager.cc:1939] T e8ea6306a7c340f1b36786584552ed6b P 16bd5000dd1a488997d78ba43b14ae16: Deleting consensus metadata
I20260810 19:23:11.438408 865 catalog_manager.cc:4841] TS 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): tablet e8ea6306a7c340f1b36786584552ed6b (table test-table [id=192eb9a7718742a6bc47fbd06abf0d8c]) successfully deleted
I20260810 19:23:11.438871 1716 ts_tablet_manager.cc:1918] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:23:11.439067 1716 log.cc:1196] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/e8ea6306a7c340f1b36786584552ed6b
I20260810 19:23:11.439634 1716 ts_tablet_manager.cc:1939] T e8ea6306a7c340f1b36786584552ed6b P 7a1386f40f734df89b4135cf7a34b356: Deleting consensus metadata
I20260810 19:23:11.440698 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.404563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.7.43.1:41089)
0810 19:23:11.404754 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.404761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.414170 (+ 9409us) server_negotiation.cc:407] Connection header received
0810 19:23:11.417095 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.417102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.417189 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.417263 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:11.420073 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.421217 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.425459 (+ 4242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.426133 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.426271 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.432822 (+ 6551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:11.432838 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:11.432857 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:11.432902 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:11.436680 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.437873 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.437883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.437887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.438047 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.438546 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.438551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.438554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.439025 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:11.439322 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.440072 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.440311 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:23:11.441641 865 catalog_manager.cc:4841] TS 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): tablet e8ea6306a7c340f1b36786584552ed6b (table test-table [id=192eb9a7718742a6bc47fbd06abf0d8c]) successfully deleted
I20260810 19:23:11.442013 1187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:11.593130 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 995
I20260810 19:23:11.618453 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 1212
Aug 10 19:23:11 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:23:11.635618 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.619272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.7.43.2:35729)
0810 19:23:11.619454 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.619460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.619544 (+ 84us) server_negotiation.cc:407] Connection header received
0810 19:23:11.619732 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.619737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.619827 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.619916 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:11.621226 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.622078 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.623449 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.623850 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.623962 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.628657 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:11.628680 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:11.628703 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:11.628758 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:11.632593 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.633346 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.633355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.633359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.633554 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.633927 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.633932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.633935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.634427 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:11.634711 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.635055 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.635277 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:23:11.636271 1411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:11.725165 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 1212
I20260810 19:23:11.746960 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 1427
Aug 10 19:23:11 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:23:11.764328 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:11.747717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.7.43.3:35497)
0810 19:23:11.747901 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:11.747907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:11.748057 (+ 150us) server_negotiation.cc:407] Connection header received
0810 19:23:11.748265 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:11.748269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:11.748337 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:11.748425 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:11.749742 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.750585 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.751963 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:11.752445 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:11.752562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:11.757190 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:11.757214 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:11.757237 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:11.757298 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:11.761141 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.761952 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.761963 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.761970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.762200 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:11.762558 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:11.762563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:11.762567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:11.763055 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:11.763416 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:11.763617 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 19:23:11.763916 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:23:11.765133 1637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:11.886528 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 1427
I20260810 19:23:11.912585 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 830
I20260810 19:23:11.913738 932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:12.027863 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 830
I20260810 19:23:32.058609 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36245
--webserver_interface=127.7.43.62
--webserver_port=41023
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:32.405932 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:32.406261 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:32.406407 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:32.413877 1726 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:23:32.413998 1726 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:23:32.414064 1726 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:23:32.414132 1726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:23:32.414209 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:32.414306 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:32.414368 1726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:23:32.414453 1726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:23:32.424566 1726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=41023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:32.426046 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:32.427434 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:32.452400 1734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:32.452443 1741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:32.452353 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:32.453313 1726 server_base.cc:1034] running on GCE node
I20260810 19:23:32.454332 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:32.456022 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:32.457273 1726 hybrid_clock.cc:648] HybridClock initialized: now 1786389812457195 us; error 51 us; skew 500 ppm
I20260810 19:23:32.457701 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:32 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389812, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:32.464556 1726 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:23:32.466351 1726 webserver.cc:460] Webserver started at http://127.7.43.62:41023/ using document root <none> and password file <none>
I20260810 19:23:32.467177 1726 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:32.467310 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:32.473690 1726 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:23:32.476825 1752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:32.478332 1726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:23:32.478528 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "07730a3a82f54f2b94153c1184a50747"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "c13cce55b31b45f6b2da569b1e6dbd02"
server_key_iv: "4b029c7cfa66de8a6a6ad5ce144fa9fe"
server_key_version: "encryptionkey@0"
I20260810 19:23:32.479647 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:32.492656 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:32.497082 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:32.497383 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:32.516481 1726 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:36245
I20260810 19:23:32.516503 1838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:36245 every 8 connection(s)
I20260810 19:23:32.518148 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:23:32.525481 1839 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:23:32.528788 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 1726
I20260810 19:23:32.529320 1839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap starting.
I20260810 19:23:32.529942 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:41089
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:36245
--webserver_port=44345
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:23:32.552685 1839 log.cc:824] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:32.565680 1839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:23:32.566256 1839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap complete.
I20260810 19:23:32.575623 1839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:32.576522 1839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07730a3a82f54f2b94153c1184a50747, State: Initialized, Role: FOLLOWER
I20260810 19:23:32.577216 1839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:32.577453 1839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:23:32.577636 1839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:23:32.577805 1839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:23:32.581719 1839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:32.582242 1839 leader_election.cc:302] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07730a3a82f54f2b94153c1184a50747; no voters:
I20260810 19:23:32.582880 1839 leader_election.cc:288] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:23:32.583015 1849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:23:32.584237 1849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 LEADER]: Becoming Leader. State: Replica: 07730a3a82f54f2b94153c1184a50747, State: Running, Role: LEADER
I20260810 19:23:32.584880 1849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:32.585451 1839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:23:32.587463 1851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07730a3a82f54f2b94153c1184a50747. Latest consensus state: current_term: 2 leader_uuid: "07730a3a82f54f2b94153c1184a50747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } } }
I20260810 19:23:32.587872 1851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: This master's current role is: LEADER
I20260810 19:23:32.588434 1850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07730a3a82f54f2b94153c1184a50747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } } }
I20260810 19:23:32.588740 1850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: This master's current role is: LEADER
I20260810 19:23:32.594252 1854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:23:32.603410 1854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:23:32.604914 1854 catalog_manager.cc:1259] Loaded cluster ID: 22755aa2e51d45de8e70bbbe23a5da3c
I20260810 19:23:32.605085 1854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:23:32.609129 1854 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:23:32.610782 1854 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Loaded TSK: 0
I20260810 19:23:32.618108 1875 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:23:32.649844 1854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Generated new TSK 1
I20260810 19:23:32.650283 1854 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:23:32.905755 1841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:32.906086 1841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:32.906221 1841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:32.913611 1841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:32.913738 1841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:32.913920 1841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:23:32.924185 1841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:41089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=44345
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:32.925504 1841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:32.926899 1841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:32.941605 1884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:32.942137 1888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:32.942202 1890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:32.943292 1841 server_base.cc:1034] running on GCE node
I20260810 19:23:32.944021 1841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:32.945019 1841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:32.946233 1841 hybrid_clock.cc:648] HybridClock initialized: now 1786389812946173 us; error 52 us; skew 500 ppm
I20260810 19:23:32.946545 1841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:32 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389812, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:32.949851 1841 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:23:32.951411 1841 webserver.cc:460] Webserver started at http://127.7.43.1:44345/ using document root <none> and password file <none>
I20260810 19:23:32.952109 1841 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:32.952244 1841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:32.957837 1841 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 19:23:32.960737 1904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:32.962270 1841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:23:32.962432 1841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "8dc9fbe8df0847d985170952e5932353"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "82e5adcff0c4b886871813e9007b3188"
server_key_iv: "baf3265c3f53899ce9b74f881ae7d61b"
server_key_version: "encryptionkey@0"
I20260810 19:23:32.963445 1841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:32.978219 1841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:32.982026 1841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:32.982286 1841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:32.983309 1841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:32.984680 1841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:32.984783 1841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:32.984892 1841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:32.984979 1841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:33.015862 1841 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:41089
I20260810 19:23:33.015923 2092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:41089 every 8 connection(s)
I20260810 19:23:33.017589 1841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:23:33.026288 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 1841
I20260810 19:23:33.026943 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:35729
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:36245
--webserver_port=36349
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:23:33 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389812, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:33.042524 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.020859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:52167)
0810 19:23:33.021709 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:33.021763 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:33.023560 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:33.024871 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:33.024878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:33.025414 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:33.026252 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:33.026284 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.028019 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.028024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:33.029878 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:33.029893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.031069 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.031076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:33.031338 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.032051 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:33.032080 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:33.034657 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:33.038087 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:33.038097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:33.038114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:33.038539 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:33.038975 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:33.038980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:33.038983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:33.039180 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:33.039934 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:33.039942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:33.040282 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 19:23:33.040525 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 19:23:33.041127 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":318,"threads_started":1}
I20260810 19:23:33.042522 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.020694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:52167 (local address 127.7.43.62:36245)
0810 19:23:33.021281 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:33.021294 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:33.021902 (+ 608us) server_negotiation.cc:407] Connection header received
0810 19:23:33.024070 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:33.024084 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:33.024431 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:33.024734 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:33.026482 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.027836 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.030014 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.030804 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.030944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.034950 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:33.034959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:33.034975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:33.035019 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:33.037931 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.038686 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.038694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.038700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.038863 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.039305 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.039308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.039310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.039794 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:33.039991 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:33.040848 (+ 857us) server_negotiation.cc:299] Negotiation successful
0810 19:23:33.041264 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":247,"threads_started":1}
I20260810 19:23:33.045627 2093 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36245
I20260810 19:23:33.046029 2093 heartbeater.cc:464] Registering TS with master...
I20260810 19:23:33.046969 2093 heartbeater.cc:511] Master 127.7.43.62:36245 requested a full tablet report, sending...
I20260810 19:23:33.049415 1781 ts_manager.cc:194] Registered new tserver with Master: 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089)
I20260810 19:23:33.051026 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:52167
W20260810 19:23:33.377586 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:33.377933 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:33.378093 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:33.385722 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:33.385885 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:33.386057 2097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:23:33.396396 2097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:35729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=36349
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:33.397843 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:33.399245 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:33.413296 2110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:33.413226 2109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:33.413264 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:33.414101 2097 server_base.cc:1034] running on GCE node
I20260810 19:23:33.414740 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:33.415719 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:33.416932 2097 hybrid_clock.cc:648] HybridClock initialized: now 1786389813416885 us; error 35 us; skew 500 ppm
I20260810 19:23:33.417255 2097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:33 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389813, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:33.420452 2097 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:23:33.421996 2097 webserver.cc:460] Webserver started at http://127.7.43.2:36349/ using document root <none> and password file <none>
I20260810 19:23:33.422688 2097 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:33.422820 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:33.428212 2097 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:23:33.430970 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:33.432331 2097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 19:23:33.432487 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "16bd5000dd1a488997d78ba43b14ae16"
format_stamp: "Formatted at 2026-08-10 19:23:10 on dist-test-slave-7lbf"
server_key: "0a9f6290caaa8ddd5db2f8116b8a5cd0"
server_key_iv: "c0658f06854dfc2dc7a2577f61e5161b"
server_key_version: "encryptionkey@0"
I20260810 19:23:33.433457 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:33.448484 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:33.452646 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:33.452906 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:33.454139 2097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:33.455627 2097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:33.455716 2097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:33.455797 2097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:33.455853 2097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:33.487753 2097 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:35729
I20260810 19:23:33.487815 2298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:35729 every 8 connection(s)
I20260810 19:23:33.489364 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:23:33.490583 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 2097
I20260810 19:23:33.491385 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:35497
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:36245
--webserver_port=37587
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:23:33 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389813, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:33.512701 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.492455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:58729 (local address 127.7.43.62:36245)
0810 19:23:33.492679 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:33.492685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:33.494020 (+ 1335us) server_negotiation.cc:407] Connection header received
0810 19:23:33.496408 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:33.496412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:33.496486 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:33.496584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:33.498599 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.499282 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.501343 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.501886 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.501984 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.506349 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:33.506356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:33.506360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:33.506402 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:33.509423 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.510250 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.510256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.510259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.510411 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.510864 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.510869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.510872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.511233 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:33.511371 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:33.512141 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 19:23:33.512373 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:23:33.513612 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.492936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:58729)
0810 19:23:33.493822 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:33.493881 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:33.496099 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:33.496860 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:33.496872 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:33.497625 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:33.498417 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:33.498445 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.499447 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.499452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:33.501214 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:33.501230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.502110 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.502117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:33.502408 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.503147 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:33.503176 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:33.506029 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:33.509618 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:33.509627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:33.509642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:33.510104 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:33.510512 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:33.510515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:33.510518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:33.510741 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:33.511350 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:33.511356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:33.511752 (+ 396us) client_negotiation.cc:769] Sending connection context
0810 19:23:33.511988 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 19:23:33.512401 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":312,"threads_started":1}
I20260810 19:23:33.515132 2299 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36245
I20260810 19:23:33.515462 2299 heartbeater.cc:464] Registering TS with master...
I20260810 19:23:33.516266 2299 heartbeater.cc:511] Master 127.7.43.62:36245 requested a full tablet report, sending...
I20260810 19:23:33.517825 1781 ts_manager.cc:194] Registered new tserver with Master: 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729)
I20260810 19:23:33.518698 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:58729
W20260810 19:23:33.841384 2303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:33.841766 2303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:33.841905 2303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:33.849653 2303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:33.849776 2303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:33.849923 2303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:23:33.860241 2303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:35497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=37587
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:33.861666 2303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:33.863059 2303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:33.877077 2313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:33.877097 2308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:33.877137 2309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:33.878147 2303 server_base.cc:1034] running on GCE node
I20260810 19:23:33.878806 2303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:33.879817 2303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:33.881027 2303 hybrid_clock.cc:648] HybridClock initialized: now 1786389813880968 us; error 50 us; skew 500 ppm
I20260810 19:23:33.881377 2303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:33 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389813, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:33.884799 2303 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:23:33.886447 2303 webserver.cc:460] Webserver started at http://127.7.43.3:37587/ using document root <none> and password file <none>
I20260810 19:23:33.887224 2303 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:33.887351 2303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:33.893095 2303 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:23:33.896111 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:33.897564 2303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:23:33.897732 2303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "7a1386f40f734df89b4135cf7a34b356"
format_stamp: "Formatted at 2026-08-10 19:23:11 on dist-test-slave-7lbf"
server_key: "b6c009da7f2d85691e28612c17df871d"
server_key_iv: "50fa4f098fe46e7a088d76b4097452d8"
server_key_version: "encryptionkey@0"
I20260810 19:23:33.898756 2303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:33.918792 2303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:33.922782 2303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:33.923065 2303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:33.924079 2303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:33.925491 2303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:33.925616 2303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 19:23:33.925750 2303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:33.925827 2303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:33.957422 2303 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:35497
I20260810 19:23:33.957492 2507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:35497 every 8 connection(s)
I20260810 19:23:33.958941 2303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:23:33.967411 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 2303
Aug 10 19:23:33 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389813, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:23:33.982240 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.961611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:43457 (local address 127.7.43.62:36245)
0810 19:23:33.961796 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:33.961801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:33.962966 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 19:23:33.964856 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:33.964862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:33.964947 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:33.965049 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:33.966956 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.967815 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.969988 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.970587 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.970721 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.975176 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:33.975181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:33.975184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:33.975221 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:33.978751 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.979694 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.979698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.979701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.979829 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.980280 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.980284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.980286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.980603 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:33.980736 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:33.981580 (+ 844us) server_negotiation.cc:299] Negotiation successful
0810 19:23:33.981864 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:23:33.983495 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.962071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:43457)
0810 19:23:33.962797 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:33.962839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:33.964559 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:33.965320 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:33.965329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:33.966076 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:33.966744 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:33.966789 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.968054 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.968059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:33.969840 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:33.969856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.970900 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.970907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:33.971141 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.971957 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:33.971990 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:33.974916 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:33.978904 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:33.978912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:33.978927 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:33.979560 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:33.979945 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:33.979949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:33.979951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:33.980150 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:33.980737 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:33.980746 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:33.981124 (+ 378us) client_negotiation.cc:769] Sending connection context
0810 19:23:33.981384 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 19:23:33.982014 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":273,"threads_started":1}
I20260810 19:23:33.984774 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:33.968410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.7.43.62:36245)
0810 19:23:33.969005 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:33.969013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:33.970096 (+ 1083us) server_negotiation.cc:407] Connection header received
0810 19:23:33.970294 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:33.970307 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:33.970379 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:33.970484 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:33.972352 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.973476 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.975099 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:33.975620 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:33.975739 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:33.979154 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:33.979161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:33.979165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:33.979204 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:33.981881 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.982619 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.982625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.982628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.982770 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:33.983182 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:33.983186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:33.983189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:33.983585 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:33.983838 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:33.984193 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 19:23:33.984420 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":296,"threads_started":1}
I20260810 19:23:33.985374 2508 heartbeater.cc:347] Connected to a master server at 127.7.43.62:36245
I20260810 19:23:33.985713 2508 heartbeater.cc:464] Registering TS with master...
I20260810 19:23:33.986464 2508 heartbeater.cc:511] Master 127.7.43.62:36245 requested a full tablet report, sending...
I20260810 19:23:33.987941 1781 ts_manager.cc:194] Registered new tserver with Master: 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
I20260810 19:23:33.988772 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:43457
I20260810 19:23:33.999929 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:23:34.006870 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.000934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.7.43.62:36245)
0810 19:23:34.001191 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.001197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.001471 (+ 274us) server_negotiation.cc:407] Connection header received
0810 19:23:34.001719 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.001729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.001900 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.002027 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:23:34.002509 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.003311 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.004924 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.005332 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.005424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.005977 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:23:34.006285 (+ 308us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 19:23:34.006618 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42940: token expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 19:23:34.007028 2514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42940: token expired
W20260810 19:23:34.007629 1651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.43.62:36245)
I20260810 19:23:34.020967 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.008178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.7.43.62:36245)
0810 19:23:34.008350 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.008354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.008409 (+ 55us) server_negotiation.cc:407] Connection header received
0810 19:23:34.008540 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.008544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.008602 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.008668 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:34.010207 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.010952 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.012237 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.012639 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.012731 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.015447 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:34.015453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:34.015456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:34.015491 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:34.018338 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.019028 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.019034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.019038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.019169 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.019558 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.019562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.019564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.019891 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:34.020121 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.020425 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.020637 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:23:34.024609 1781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:23:34.032286 1781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:23:34.055071 2093 heartbeater.cc:503] Master 127.7.43.62:36245 was elected leader, sending a full tablet report...
I20260810 19:23:34.055896 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.045692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35729 (local address 127.0.0.1:51126)
0810 19:23:34.046262 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.046282 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:34.046409 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:34.047420 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:34.047425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:34.047452 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:34.047824 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.047838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.050043 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.050049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.052728 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.052748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.054739 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.054746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.054943 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.055003 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 19:23:34.055121 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 19:23:34.055447 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":53,"thread_start_us":283,"threads_started":1}
I20260810 19:23:34.057180 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.045937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.7.43.2:35729)
0810 19:23:34.046906 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.046913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.046934 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:23:34.047005 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.047016 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.047201 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.048035 (+ 834us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:34.048274 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.049886 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.052926 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.054479 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.056351 (+ 1872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.056575 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.056662 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.056835 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":696,"threads_started":1}
I20260810 19:23:34.057992 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.044292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59774)
0810 19:23:34.045234 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.045277 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:34.045473 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:34.046633 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:34.046641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:34.046672 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:34.047040 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.047053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.050449 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.050455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.054760 (+ 4305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.054775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.057095 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.057100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.057274 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.057317 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 19:23:34.057450 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 19:23:34.057616 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":537,"threads_started":1}
I20260810 19:23:34.058729 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.044602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.7.43.1:41089)
0810 19:23:34.045242 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.045251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.045560 (+ 309us) server_negotiation.cc:407] Connection header received
0810 19:23:34.045800 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.045810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.046103 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.046289 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:34.047234 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.050297 (+ 3063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.054978 (+ 4681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.056525 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.056623 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.056903 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.058229 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.058424 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":684,"threads_started":1}
I20260810 19:23:34.062263 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.046329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:35497 (local address 127.0.0.1:56944)
0810 19:23:34.054467 (+ 8138us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.054488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:34.054632 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:34.055659 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:34.055664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:34.055696 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:34.056068 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.056081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.058187 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.058192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.059733 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.059750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.061455 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.061462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.061666 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.061699 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:23:34.061796 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:23:34.061937 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7992,"thread_start_us":7839,"threads_started":1}
I20260810 19:23:34.062505 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.046905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.7.43.3:35497)
0810 19:23:34.047664 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.047673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.054614 (+ 6941us) server_negotiation.cc:407] Connection header received
0810 19:23:34.054797 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.054806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.055103 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.055306 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:34.056421 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.057995 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.059891 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.061233 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.061365 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.061632 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.061854 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.062012 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":419,"threads_started":1}
I20260810 19:23:34.061959 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 828a1a37a74b4aadb0203100723cfcfa (DEFAULT_TABLE table=test-table [id=f2481e8798d545e08fd2d3a774b52a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:23:34.061673 2202 tablet_service.cc:1467] Processing CreateTablet for tablet 828a1a37a74b4aadb0203100723cfcfa (DEFAULT_TABLE table=test-table [id=f2481e8798d545e08fd2d3a774b52a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:23:34.063647 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828a1a37a74b4aadb0203100723cfcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:34.064033 2202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828a1a37a74b4aadb0203100723cfcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:34.066198 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 828a1a37a74b4aadb0203100723cfcfa (DEFAULT_TABLE table=test-table [id=f2481e8798d545e08fd2d3a774b52a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:23:34.067658 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828a1a37a74b4aadb0203100723cfcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:23:34.077823 2533 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Bootstrap starting.
I20260810 19:23:34.079943 2534 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Bootstrap starting.
I20260810 19:23:34.081878 2533 tablet_bootstrap.cc:654] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:34.082978 2535 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Bootstrap starting.
I20260810 19:23:34.083554 2534 tablet_bootstrap.cc:654] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:34.083710 2533 log.cc:824] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:34.084846 2534 log.cc:824] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:34.086726 2535 tablet_bootstrap.cc:654] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Neither blocks nor log segments found. Creating new log.
I20260810 19:23:34.087666 2535 log.cc:824] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:34.087827 2533 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: No bootstrap required, opened a new log
I20260810 19:23:34.088141 2533 ts_tablet_manager.cc:1397] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260810 19:23:34.088325 2534 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: No bootstrap required, opened a new log
I20260810 19:23:34.088596 2534 ts_tablet_manager.cc:1397] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 19:23:34.090193 2535 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: No bootstrap required, opened a new log
I20260810 19:23:34.090431 2535 ts_tablet_manager.cc:1397] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:23:34.098111 2535 raft_consensus.cc:348] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.098513 2535 raft_consensus.cc:374] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:34.098620 2535 raft_consensus.cc:729] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1386f40f734df89b4135cf7a34b356, State: Initialized, Role: FOLLOWER
I20260810 19:23:34.099257 2535 consensus_queue.cc:260] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.099165 2534 raft_consensus.cc:348] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.099165 2533 raft_consensus.cc:348] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.099610 2534 raft_consensus.cc:374] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:34.099609 2533 raft_consensus.cc:374] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:23:34.099715 2533 raft_consensus.cc:729] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc9fbe8df0847d985170952e5932353, State: Initialized, Role: FOLLOWER
I20260810 19:23:34.099720 2534 raft_consensus.cc:729] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bd5000dd1a488997d78ba43b14ae16, State: Initialized, Role: FOLLOWER
I20260810 19:23:34.100458 2508 heartbeater.cc:503] Master 127.7.43.62:36245 was elected leader, sending a full tablet report...
I20260810 19:23:34.100452 2533 consensus_queue.cc:260] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.100452 2534 consensus_queue.cc:260] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.101233 2535 ts_tablet_manager.cc:1428] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 19:23:34.106282 2299 heartbeater.cc:503] Master 127.7.43.62:36245 was elected leader, sending a full tablet report...
I20260810 19:23:34.107017 2534 ts_tablet_manager.cc:1428] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
I20260810 19:23:34.108774 2533 ts_tablet_manager.cc:1428] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Time spent starting tablet: real 0.021s user 0.010s sys 0.009s
I20260810 19:23:34.181872 2543 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:23:34.182205 2543 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.184142 2543 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089), 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729)
I20260810 19:23:34.190131 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.184397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.7.43.3:43453)
0810 19:23:34.184682 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.184705 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:34.184878 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:34.185314 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:34.185319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:34.185351 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:34.185609 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.185624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.186925 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.186931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.188241 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.188254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.189364 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.189370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.189583 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.189609 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:23:34.189701 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 19:23:34.189841 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 19:23:34.190397 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.184520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:43453 (local address 127.7.43.1:41089)
0810 19:23:34.184728 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.184734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.184813 (+ 79us) server_negotiation.cc:407] Connection header received
0810 19:23:34.185087 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.185095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.185183 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.185278 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:34.185787 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.186809 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.188413 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.189200 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.189324 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.189430 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.189869 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.190067 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:23:34.191193 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.185137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:60573 (local address 127.7.43.2:35729)
0810 19:23:34.185332 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.185339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.185720 (+ 381us) server_negotiation.cc:407] Connection header received
0810 19:23:34.185846 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.185850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.185910 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.185978 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:23:34.186572 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.187313 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.188973 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.190000 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.190143 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.190244 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.190795 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.190935 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:23:34.191211 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.184850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35729 (local address 127.7.43.3:60573)
0810 19:23:34.185586 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.185604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:34.185746 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:34.186019 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:34.186024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:34.186051 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:23:34.186438 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.186450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.187445 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.187450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.188822 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:34.188834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.190377 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.190383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:34.190571 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.190606 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 19:23:34.190744 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 19:23:34.190869 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":485,"threads_started":1}
I20260810 19:23:34.192318 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a1a37a74b4aadb0203100723cfcfa" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd5000dd1a488997d78ba43b14ae16" is_pre_election: true
I20260810 19:23:34.192269 2011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a1a37a74b4aadb0203100723cfcfa" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc9fbe8df0847d985170952e5932353" is_pre_election: true
I20260810 19:23:34.192890 2230 raft_consensus.cc:2454] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 0.
I20260810 19:23:34.192893 2011 raft_consensus.cc:2454] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 0.
I20260810 19:23:34.193574 2332 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356; no voters:
I20260810 19:23:34.193956 2543 raft_consensus.cc:2781] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:23:34.194093 2543 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:23:34.194192 2543 raft_consensus.cc:3037] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:34.196480 2543 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.197116 2543 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 election: Requested vote from peers 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089), 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729)
I20260810 19:23:34.197507 2011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a1a37a74b4aadb0203100723cfcfa" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc9fbe8df0847d985170952e5932353"
I20260810 19:23:34.197583 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a1a37a74b4aadb0203100723cfcfa" candidate_uuid: "7a1386f40f734df89b4135cf7a34b356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd5000dd1a488997d78ba43b14ae16"
I20260810 19:23:34.197729 2011 raft_consensus.cc:3037] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:34.197759 2230 raft_consensus.cc:3037] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:23:34.200002 2011 raft_consensus.cc:2454] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 1.
I20260810 19:23:34.200161 2230 raft_consensus.cc:2454] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a1386f40f734df89b4135cf7a34b356 in term 1.
I20260810 19:23:34.200481 2332 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a1386f40f734df89b4135cf7a34b356, 8dc9fbe8df0847d985170952e5932353; no voters:
I20260810 19:23:34.200778 2543 raft_consensus.cc:2781] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:23:34.201215 2543 raft_consensus.cc:686] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 1 LEADER]: Becoming Leader. State: Replica: 7a1386f40f734df89b4135cf7a34b356, State: Running, Role: LEADER
I20260810 19:23:34.201731 2543 consensus_queue.cc:237] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:34.206137 1780 catalog_manager.cc:5495] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "7a1386f40f734df89b4135cf7a34b356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } health_report { overall_health: HEALTHY } } }
W20260810 19:23:34.210469 2510 tablet.cc:2369] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:23:34.241472 2300 tablet.cc:2369] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:23:34.249862 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.244773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.7.43.3:35497)
0810 19:23:34.244953 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.244958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.244975 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:23:34.245078 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.245082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.245160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.245293 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:23:34.245738 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.246451 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.247819 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.248197 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.248284 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.248790 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:23:34.249298 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:23:34.249415 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.249582 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.249655 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:23:34.258179 2011 raft_consensus.cc:1261] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Refusing update from remote peer 7a1386f40f734df89b4135cf7a34b356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:23:34.258194 2230 raft_consensus.cc:1261] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Refusing update from remote peer 7a1386f40f734df89b4135cf7a34b356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:23:34.259119 2549 consensus_queue.cc:1035] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:23:34.259553 2543 consensus_queue.cc:1035] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:23:34.269942 2094 tablet.cc:2369] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:23:34.270814 2555 mvcc.cc:205] Tried to move back new op lower bound from 7317052679191658496 to 7317052678974095360. Current Snapshot: MvccSnapshot[applied={T|T < 7317052679191658496}]
I20260810 19:23:34.274277 2558 mvcc.cc:205] Tried to move back new op lower bound from 7317052679191658496 to 7317052678974095360. Current Snapshot: MvccSnapshot[applied={T|T < 7317052679191658496}]
I20260810 19:23:34.287926 1781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:23:34.288293 1781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:23:34.291963 1781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Sending DeleteTablet for 3 replicas of tablet 828a1a37a74b4aadb0203100723cfcfa
I20260810 19:23:34.293277 1983 tablet_service.cc:1514] Processing DeleteTablet for tablet 828a1a37a74b4aadb0203100723cfcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:23:34 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59774
I20260810 19:23:34.293570 2202 tablet_service.cc:1514] Processing DeleteTablet for tablet 828a1a37a74b4aadb0203100723cfcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:23:34 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:51126
I20260810 19:23:34.293638 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 828a1a37a74b4aadb0203100723cfcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:23:34 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:56944
I20260810 19:23:34.294401 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 1841
I20260810 19:23:34.305276 2575 tablet_replica.cc:330] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: stopping tablet replica
I20260810 19:23:34.305796 2575 raft_consensus.cc:2229] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:23:34.306360 2575 raft_consensus.cc:2258] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:23:34.308661 2575 ts_tablet_manager.cc:1905] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:23:34.313572 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.295961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.7.43.1:41089)
0810 19:23:34.296114 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.296121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.296139 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:23:34.296199 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.296205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.296314 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.296427 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:34.298210 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.299502 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.301467 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.302200 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.302354 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.305784 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:34.305802 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:34.305840 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:34.305895 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:34.310417 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.311313 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.311322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.311326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.311510 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.311894 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.311900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.311904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.312433 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:34.312720 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.312981 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.313188 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 19:23:34.314558 2083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:34.315434 2576 tablet_replica.cc:330] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: stopping tablet replica
I20260810 19:23:34.316066 2576 raft_consensus.cc:2229] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:23:34.316905 2576 raft_consensus.cc:2258] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:23:34.319398 2576 ts_tablet_manager.cc:1905] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:23:34.322032 2575 ts_tablet_manager.cc:1918] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:23:34.322220 2575 log.cc:1196] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/828a1a37a74b4aadb0203100723cfcfa
I20260810 19:23:34.322781 2575 ts_tablet_manager.cc:1939] T 828a1a37a74b4aadb0203100723cfcfa P 16bd5000dd1a488997d78ba43b14ae16: Deleting consensus metadata
I20260810 19:23:34.324815 1758 catalog_manager.cc:4841] TS 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): tablet 828a1a37a74b4aadb0203100723cfcfa (table test-table [id=f2481e8798d545e08fd2d3a774b52a83]) successfully deleted
I20260810 19:23:34.328271 2576 ts_tablet_manager.cc:1918] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:23:34.328541 2576 log.cc:1196] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/828a1a37a74b4aadb0203100723cfcfa
I20260810 19:23:34.329257 2576 ts_tablet_manager.cc:1939] T 828a1a37a74b4aadb0203100723cfcfa P 7a1386f40f734df89b4135cf7a34b356: Deleting consensus metadata
I20260810 19:23:34.331422 1758 catalog_manager.cc:4841] TS 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): tablet 828a1a37a74b4aadb0203100723cfcfa (table test-table [id=f2481e8798d545e08fd2d3a774b52a83]) successfully deleted
I20260810 19:23:34.418430 2574 tablet_replica.cc:330] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: stopping tablet replica
I20260810 19:23:34.418623 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 1841
W20260810 19:23:34.450675 1758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:41089 (error 108)
I20260810 19:23:34.450896 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 2097
I20260810 19:23:34.453099 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.452680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59794)
0810 19:23:34.452865 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.452953 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:23:34.453352 1758 catalog_manager.cc:4584] TS 8dc9fbe8df0847d985170952e5932353 (127.7.43.1:41089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 828a1a37a74b4aadb0203100723cfcfa: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
I20260810 19:23:34.466380 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.451804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.7.43.2:35729)
0810 19:23:34.451993 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.451999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.452023 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:23:34.452227 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.452233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.452331 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.452448 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:34.454232 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.454996 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.456880 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.457257 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.457345 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.460612 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:34.460627 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:34.460643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:34.460681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:34.463467 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.464250 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.464256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.464258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.464384 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.464756 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.464759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.464761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.465135 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:34.465355 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.465950 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.466115 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:23:34.467197 2293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:34.507122 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.506567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59796)
0810 19:23:34.506814 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.506934 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 19:23:34.508322 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.507904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59800)
0810 19:23:34.508070 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.508161 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 19:23:34.569280 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.568644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59814)
0810 19:23:34.568957 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.569067 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 19:23:34.569584 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 2097
I20260810 19:23:34.576498 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.575942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59826)
0810 19:23:34.576213 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.576320 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 19:23:34.591892 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 2303
I20260810 19:23:34.608480 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.592877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.7.43.3:35497)
0810 19:23:34.593070 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:34.593076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:34.593286 (+ 210us) server_negotiation.cc:407] Connection header received
0810 19:23:34.593474 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:34.593479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:34.593612 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:34.593716 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:34.595656 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.596554 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.598446 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:34.598887 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:34.598981 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:34.602266 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:34.602280 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:34.602297 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:34.602346 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:34.605307 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.606314 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.606322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.606324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.606514 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:34.606971 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:34.606974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:34.606976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:34.607348 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:34.607577 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:34.607955 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:23:34.608151 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:23:34.609220 2497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:34.657305 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.656769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59842)
0810 19:23:34.657002 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.657120 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 19:23:34.658468 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:34.658060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:41089 (local address 127.0.0.1:59848)
0810 19:23:34.658275 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:34.658348 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:41089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 19:23:34.733932 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 2303
I20260810 19:23:34.760835 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 1726
I20260810 19:23:34.761983 1833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:23:34.877703 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 1726
I20260810 19:23:54.906205 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36245
--webserver_interface=127.7.43.62
--webserver_port=41023
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:55.281913 2585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:55.282205 2585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:55.282336 2585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:55.289621 2585 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:23:55.289721 2585 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:23:55.289767 2585 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:23:55.289844 2585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:23:55.289922 2585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:55.290012 2585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:55.290072 2585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:23:55.290129 2585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:23:55.300096 2585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=41023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:55.301426 2585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:55.302858 2585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:55.315119 2594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:55.315055 2599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:55.315431 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:55.315866 2585 server_base.cc:1034] running on GCE node
I20260810 19:23:55.316670 2585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:55.318378 2585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:55.319626 2585 hybrid_clock.cc:648] HybridClock initialized: now 1786389835319524 us; error 51 us; skew 500 ppm
I20260810 19:23:55.320078 2585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:55 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389835, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:55.328388 2585 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:23:55.330176 2585 webserver.cc:460] Webserver started at http://127.7.43.62:41023/ using document root <none> and password file <none>
I20260810 19:23:55.330971 2585 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:55.331079 2585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:55.336877 2585 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 19:23:55.340095 2613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:55.341565 2585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:23:55.341753 2585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "07730a3a82f54f2b94153c1184a50747"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "c13cce55b31b45f6b2da569b1e6dbd02"
server_key_iv: "4b029c7cfa66de8a6a6ad5ce144fa9fe"
server_key_version: "encryptionkey@0"
I20260810 19:23:55.342867 2585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:55.376227 2585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:55.381033 2585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:55.381320 2585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:55.400580 2585 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:36245
I20260810 19:23:55.400637 2701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:36245 every 8 connection(s)
I20260810 19:23:55.402390 2585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:23:55.410050 2704 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:23:55.410897 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 2585
I20260810 19:23:55.411990 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:41089
--local_ip_for_outbound_sockets=127.7.43.1
--tserver_master_addrs=127.7.43.62:36245
--webserver_port=44345
--webserver_interface=127.7.43.1
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:23:55.415136 2704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap starting.
I20260810 19:23:55.439061 2704 log.cc:824] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:55.456544 2704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:23:55.457101 2704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Bootstrap complete.
I20260810 19:23:55.466456 2704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:55.467383 2704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07730a3a82f54f2b94153c1184a50747, State: Initialized, Role: FOLLOWER
I20260810 19:23:55.468107 2704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:55.468385 2704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:23:55.468521 2704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:23:55.468698 2704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 2 FOLLOWER]: Advancing to term 3
I20260810 19:23:55.472663 2704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:55.473165 2704 leader_election.cc:302] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07730a3a82f54f2b94153c1184a50747; no voters:
I20260810 19:23:55.473901 2704 leader_election.cc:288] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 19:23:55.474037 2709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 19:23:55.475245 2709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [term 3 LEADER]: Becoming Leader. State: Replica: 07730a3a82f54f2b94153c1184a50747, State: Running, Role: LEADER
I20260810 19:23:55.475890 2709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } }
I20260810 19:23:55.476181 2704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:23:55.478478 2711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07730a3a82f54f2b94153c1184a50747. Latest consensus state: current_term: 3 leader_uuid: "07730a3a82f54f2b94153c1184a50747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } } }
I20260810 19:23:55.478881 2711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: This master's current role is: LEADER
I20260810 19:23:55.479449 2710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "07730a3a82f54f2b94153c1184a50747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07730a3a82f54f2b94153c1184a50747" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 36245 } } }
I20260810 19:23:55.479732 2710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747 [sys.catalog]: This master's current role is: LEADER
I20260810 19:23:55.484479 2719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:23:55.493959 2719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:23:55.495682 2719 catalog_manager.cc:1259] Loaded cluster ID: 22755aa2e51d45de8e70bbbe23a5da3c
I20260810 19:23:55.495862 2719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:23:55.500820 2719 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:23:55.502770 2719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Loaded TSK: 0
I20260810 19:23:55.503019 2719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07730a3a82f54f2b94153c1184a50747: Loaded TSK: 1
F20260810 19:23:55.504346 2719 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcb162d29ad google::LogMessage::Fail() at ??:0
@ 0x7fcb162d588a google::LogMessage::SendToLog() at ??:0
@ 0x7fcb162d24de google::LogMessage::Flush() at ??:0
@ 0x7fcb162d3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcb1a0367d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcb1a02573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcb2bd011c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcb2bd00783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcb2bd8c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcb2bd8c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcb2bdd65e9 std::function<>::operator()() at ??:0
@ 0x7fcb2bd04cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcb2bd02c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcb2bd8a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcb2bd8a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcb18f1e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcb18f2191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcb18f21799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcb18efceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcb1c1d66db start_thread at ??:0
@ 0x7fcb1516c61f clone at ??:0
W20260810 19:23:55.810034 2706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:55.810338 2706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:55.810480 2706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:55.817973 2706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:55.818107 2706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:55.818269 2706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:23:55.828258 2706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:41089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=44345
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:55.829596 2706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:55.830994 2706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:55.844403 2788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:55.844424 2789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:55.844365 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:55.845240 2706 server_base.cc:1034] running on GCE node
I20260810 19:23:55.845994 2706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:55.847019 2706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:55.848248 2706 hybrid_clock.cc:648] HybridClock initialized: now 1786389835848188 us; error 48 us; skew 500 ppm
I20260810 19:23:55.848598 2706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:55 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389835, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:55.852044 2706 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:23:55.853734 2706 webserver.cc:460] Webserver started at http://127.7.43.1:44345/ using document root <none> and password file <none>
I20260810 19:23:55.854486 2706 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:55.854615 2706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:55.860509 2706 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 19:23:55.863713 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:55.865201 2706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:23:55.865367 2706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "8dc9fbe8df0847d985170952e5932353"
format_stamp: "Formatted at 2026-08-10 19:23:09 on dist-test-slave-7lbf"
server_key: "82e5adcff0c4b886871813e9007b3188"
server_key_iv: "baf3265c3f53899ce9b74f881ae7d61b"
server_key_version: "encryptionkey@0"
I20260810 19:23:55.866473 2706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:55.882809 2706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:55.887205 2706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:55.887485 2706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:55.888556 2706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:55.890563 2814 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 19:23:55.896440 2706 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 19:23:55.896595 2706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260810 19:23:55.896730 2706 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 19:23:55.900098 2706 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 19:23:55.900190 2706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260810 19:23:55.900574 2814 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Bootstrap starting.
I20260810 19:23:55.928337 2814 log.cc:824] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Log is configured to *not* fsync() on all Append() calls
I20260810 19:23:55.937024 2706 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:41089
I20260810 19:23:55.937084 2967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:41089 every 8 connection(s)
I20260810 19:23:55.938375 2706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:23:55.941859 2814 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:23:55.942636 2814 tablet_bootstrap.cc:492] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Bootstrap complete.
I20260810 19:23:55.943746 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:55.941227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:55271)
0810 19:23:55.942210 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:55.942503 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":379,"threads_started":1}
I20260810 19:23:55.944219 2814 ts_tablet_manager.cc:1397] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Time spent bootstrapping tablet: real 0.044s user 0.035s sys 0.005s
W20260810 19:23:55.944384 2969 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:36245 (0 consecutive failures): Network error: Failed to ping master at 127.7.43.62:36245: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
I20260810 19:23:55.945282 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:55.944901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:34345)
0810 19:23:55.945059 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:55.945147 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:23:55.946413 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:55.945983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:45651)
0810 19:23:55.946126 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:55.946232 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:23:55.946709 2969 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:23:55.948733 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 2706
I20260810 19:23:55.949540 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:35729
--local_ip_for_outbound_sockets=127.7.43.2
--tserver_master_addrs=127.7.43.62:36245
--webserver_port=36349
--webserver_interface=127.7.43.2
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:23:55.952507 2814 raft_consensus.cc:348] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:55.953735 2814 raft_consensus.cc:729] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc9fbe8df0847d985170952e5932353, State: Initialized, Role: FOLLOWER
I20260810 19:23:55.954442 2814 consensus_queue.cc:260] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:55.956035 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:55.955675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:42567)
0810 19:23:55.955828 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:55.955912 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 19:23:55.956127 2814 ts_tablet_manager.cc:1428] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260810 19:23:55.957011 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:55.956684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:38139)
0810 19:23:55.956821 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:55.956887 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:23:56.296160 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:56.296459 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:56.296559 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:56.303830 2976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:56.303946 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:56.304064 2976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:23:56.313995 2976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:35729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=36349
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:56.315274 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:56.316594 2976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:56.329927 2984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:56.329991 2985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:56.330029 2991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:56.331218 2976 server_base.cc:1034] running on GCE node
I20260810 19:23:56.331895 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:56.332911 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:56.334124 2976 hybrid_clock.cc:648] HybridClock initialized: now 1786389836334073 us; error 52 us; skew 500 ppm
I20260810 19:23:56.334448 2976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:56 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389836, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:56.337627 2976 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:23:56.339118 2976 webserver.cc:460] Webserver started at http://127.7.43.2:36349/ using document root <none> and password file <none>
I20260810 19:23:56.339805 2976 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:56.339929 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:56.345561 2976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:23:56.348407 3003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:56.349932 2976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:23:56.350100 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "16bd5000dd1a488997d78ba43b14ae16"
format_stamp: "Formatted at 2026-08-10 19:23:10 on dist-test-slave-7lbf"
server_key: "0a9f6290caaa8ddd5db2f8116b8a5cd0"
server_key_iv: "c0658f06854dfc2dc7a2577f61e5161b"
server_key_version: "encryptionkey@0"
I20260810 19:23:56.351128 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:56.372187 2976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:56.376493 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:56.376760 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:56.377856 2976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:56.379321 2976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:56.379420 2976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:56.379534 2976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:56.379614 2976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:56.411926 2976 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:35729
I20260810 19:23:56.412010 3183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:35729 every 8 connection(s)
I20260810 19:23:56.413409 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:23:56.418653 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.416658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:47667)
0810 19:23:56.417421 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.417693 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":298,"threads_started":1}
W20260810 19:23:56.419350 3184 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:36245 (0 consecutive failures): Network error: Failed to ping master at 127.7.43.62:36245: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
I20260810 19:23:56.420233 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.419889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:39449)
0810 19:23:56.420048 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.420119 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 19:23:56.421098 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.420750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:33237)
0810 19:23:56.420906 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.420980 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 19:23:56.421336 3184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:23:56.422113 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 2976
I20260810 19:23:56.422725 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:35497
--local_ip_for_outbound_sockets=127.7.43.3
--tserver_master_addrs=127.7.43.62:36245
--webserver_port=37587
--webserver_interface=127.7.43.3
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:23:56.778506 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:23:56.778807 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:23:56.778947 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:23:56.786590 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:23:56.786739 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:23:56.786940 3188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:23:56.798969 3188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:35497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=37587
--tserver_master_addrs=127.7.43.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:23:56.800283 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:23:56.801605 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:23:56.814810 3195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:56.814757 3197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:23:56.814716 3204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:23:56.815642 3188 server_base.cc:1034] running on GCE node
I20260810 19:23:56.816248 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:23:56.817200 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:23:56.818414 3188 hybrid_clock.cc:648] HybridClock initialized: now 1786389836818365 us; error 35 us; skew 500 ppm
I20260810 19:23:56.818744 3188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:23:56 dist-test-slave-7lbf krb5kdc[807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389836, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:23:56.821849 3188 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:23:56.823238 3188 webserver.cc:460] Webserver started at http://127.7.43.3:37587/ using document root <none> and password file <none>
I20260810 19:23:56.823913 3188 fs_manager.cc:359] Metadata directory not provided
I20260810 19:23:56.824026 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:23:56.829377 3188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:23:56.832268 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:56.833673 3188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:23:56.833844 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "7a1386f40f734df89b4135cf7a34b356"
format_stamp: "Formatted at 2026-08-10 19:23:11 on dist-test-slave-7lbf"
server_key: "b6c009da7f2d85691e28612c17df871d"
server_key_iv: "50fa4f098fe46e7a088d76b4097452d8"
server_key_version: "encryptionkey@0"
I20260810 19:23:56.834865 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:23:56.864691 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:23:56.869176 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:23:56.869454 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:23:56.870586 3188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:23:56.871975 3188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:23:56.872063 3188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:56.872164 3188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:23:56.872229 3188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:23:56.903744 3188 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:35497
I20260810 19:23:56.903820 3389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:35497 every 8 connection(s)
I20260810 19:23:56.905246 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:23:56.905846 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 3188
I20260810 19:23:56.911154 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.909314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:39605)
0810 19:23:56.910049 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.910252 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
W20260810 19:23:56.911755 3392 heartbeater.cc:644] Failed to heartbeat to 127.7.43.62:36245 (0 consecutive failures): Network error: Failed to ping master at 127.7.43.62:36245: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
I20260810 19:23:56.912520 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.912212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:47967)
0810 19:23:56.912357 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.912418 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 19:23:56.913273 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.912980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:44177)
0810 19:23:56.913106 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.913175 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:23:56.913483 3392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:23:56.958611 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:56.957811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:50167)
0810 19:23:56.958290 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:56.958399 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":227,"threads_started":1}
I20260810 19:23:57.086789 3399 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:23:57.087092 3399 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:57.088761 3399 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
Aug 10 19:23:57 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389835, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
Aug 10 19:23:57 dist-test-slave-7lbf krb5kdc[807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389835, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:23:57.111395 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.089586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:35497 (local address 127.7.43.1:46627)
0810 19:23:57.091241 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:57.091259 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:57.091386 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:57.093620 (+ 2234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:57.093632 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:57.094496 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:57.095272 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:57.095292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.097031 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.097036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:57.099133 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:57.099149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.100132 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.100137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:57.100338 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:57.100980 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:57.101014 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:57.103911 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:57.107290 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:57.107300 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:57.107317 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:57.108029 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:57.108575 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:57.108579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:57.108583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:57.108820 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:57.109753 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:57.109757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:57.110111 (+ 354us) client_negotiation.cc:769] Sending connection context
0810 19:23:57.110352 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 19:23:57.110906 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"mutex_wait_us":219,"spinlock_wait_cycles":10880,"thread_start_us":1348,"threads_started":1}
I20260810 19:23:57.111419 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.089978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:46627 (local address 127.7.43.3:35497)
0810 19:23:57.090770 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:57.090787 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:57.091383 (+ 596us) server_negotiation.cc:407] Connection header received
0810 19:23:57.092974 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:57.092984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:57.093330 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:57.093651 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:57.095478 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.096842 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.099271 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.099934 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.100080 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:57.104251 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:57.104263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:57.104283 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:57.104349 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:57.107141 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:57.108200 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:57.108210 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:57.108218 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:57.108425 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:57.108963 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:57.108967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:57.108969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:57.109632 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:57.109808 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:57.110644 (+ 836us) server_negotiation.cc:299] Negotiation successful
0810 19:23:57.111024 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":423,"threads_started":1}
I20260810 19:23:57.111429 3401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.089408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:44633 (local address 127.7.43.2:35729)
0810 19:23:57.091028 (+ 1620us) server_negotiation.cc:206] Beginning negotiation
0810 19:23:57.091044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:23:57.091086 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:23:57.092699 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:23:57.092709 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:23:57.093118 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:23:57.093440 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:23:57.095475 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.096774 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.100268 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.100919 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.101045 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:57.104462 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:23:57.104470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:23:57.104483 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:23:57.104524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:23:57.107425 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:57.108378 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:57.108386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:57.108394 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:57.108605 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:23:57.109011 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:23:57.109017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:23:57.109020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:23:57.109594 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:23:57.109778 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 19:23:57.110639 (+ 861us) server_negotiation.cc:299] Negotiation successful
0810 19:23:57.111005 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":1339,"threads_started":1}
I20260810 19:23:57.112141 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.089093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:35729 (local address 127.7.43.1:44633)
0810 19:23:57.089281 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:57.089313 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 19:23:57.089895 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:23:57.093621 (+ 3726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:23:57.093632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:23:57.094497 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:23:57.095272 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:57.095292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.096960 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.096965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:57.099133 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:23:57.099149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:23:57.101108 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:23:57.101115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:23:57.101305 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:23:57.101879 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:23:57.101901 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:23:57.104206 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:23:57.107568 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:57.107573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:57.107576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:57.108216 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:57.108732 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:23:57.108736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:23:57.108739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:23:57.108890 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:23:57.109736 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:23:57.109744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:23:57.110111 (+ 367us) client_negotiation.cc:769] Sending connection context
0810 19:23:57.110353 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 19:23:57.111055 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":376,"spinlock_wait_cycles":52864}
W20260810 19:23:57.114979 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
W20260810 19:23:57.115345 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
I20260810 19:23:57.115460 2809 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dc9fbe8df0847d985170952e5932353; no voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356
I20260810 19:23:57.115873 3399 raft_consensus.cc:2726] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:23:57.422901 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.421985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:51163)
0810 19:23:57.422498 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:57.422630 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":241,"threads_started":1}
I20260810 19:23:57.914984 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.914144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:46261)
0810 19:23:57.914656 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:57.914741 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":275,"threads_started":1}
I20260810 19:23:57.960427 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:57.959659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:33323)
0810 19:23:57.960158 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:57.960245 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260810 19:23:58.424561 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:58.423852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:52383)
0810 19:23:58.424311 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:58.424401 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260810 19:23:58.824574 3410 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:23:58.824921 3410 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:23:58.825580 3410 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
W20260810 19:23:58.826625 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
W20260810 19:23:58.826850 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
I20260810 19:23:58.826942 2809 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dc9fbe8df0847d985170952e5932353; no voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356
I20260810 19:23:58.827179 3410 raft_consensus.cc:2726] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:23:58.916846 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:58.915905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:48123)
0810 19:23:58.916490 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:58.916619 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":305,"threads_started":1}
I20260810 19:23:58.962262 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:58.961332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:47027)
0810 19:23:58.961939 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:58.962045 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":272,"threads_started":1}
I20260810 19:23:59.426276 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:59.425499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:50983)
0810 19:23:59.426019 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:59.426109 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":236,"threads_started":1}
I20260810 19:23:59.918489 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:59.917793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:35613)
0810 19:23:59.918209 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:59.918290 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260810 19:23:59.963941 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:23:59.963165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:49339)
0810 19:23:59.963630 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 19:23:59.963722 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260810 19:24:00.427994 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:00.427175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:48283)
0810 19:24:00.427644 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:00.427742 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":225,"threads_started":1}
I20260810 19:24:00.920214 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:00.919455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:36955)
0810 19:24:00.919947 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:00.920034 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":232,"threads_started":1}
I20260810 19:24:00.965845 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:00.965048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:56523)
0810 19:24:00.965576 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:00.965663 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":253,"threads_started":1}
I20260810 19:24:01.429721 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:01.428976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:56313)
0810 19:24:01.429421 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:01.429509 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260810 19:24:01.658481 3423 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:01.658699 3423 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:24:01.659409 3423 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
W20260810 19:24:01.660557 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
W20260810 19:24:01.660802 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
I20260810 19:24:01.660956 2809 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dc9fbe8df0847d985170952e5932353; no voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356
I20260810 19:24:01.661252 3423 raft_consensus.cc:2726] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:24:01.922020 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:01.921235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:50253)
0810 19:24:01.921750 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:01.921839 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260810 19:24:01.967687 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:01.966811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:58885)
0810 19:24:01.967380 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:01.967486 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":258,"threads_started":1}
I20260810 19:24:02.431495 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:02.430692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:42889)
0810 19:24:02.431201 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:02.431288 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260810 19:24:02.923718 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:02.922978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:58727)
0810 19:24:02.923436 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:02.923534 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":237,"threads_started":1}
I20260810 19:24:02.969348 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:02.968638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:45183)
0810 19:24:02.969063 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:02.969147 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260810 19:24:03.433317 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:03.432510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:39291)
0810 19:24:03.433049 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:03.433129 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":243,"threads_started":1}
I20260810 19:24:03.925611 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:03.924727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:59235)
0810 19:24:03.925245 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:03.925358 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":220,"threads_started":1}
I20260810 19:24:03.971129 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:03.970346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:35399)
0810 19:24:03.970881 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:03.970962 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":299,"threads_started":1}
I20260810 19:24:04.435065 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:04.434328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:56291)
0810 19:24:04.434767 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:04.434861 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260810 19:24:04.626637 3442 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:04.626824 3442 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:24:04.627485 3442 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
W20260810 19:24:04.628551 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
W20260810 19:24:04.628801 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
I20260810 19:24:04.628914 2809 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dc9fbe8df0847d985170952e5932353; no voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356
I20260810 19:24:04.629174 3442 raft_consensus.cc:2726] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:24:04.927290 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:04.926521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:50905)
0810 19:24:04.927025 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:04.927115 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":250,"threads_started":1}
I20260810 19:24:04.972838 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:04.972005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:40087)
0810 19:24:04.972521 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:04.972613 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":261,"threads_started":1}
I20260810 19:24:05.436817 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:05.436085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:39579)
0810 19:24:05.436570 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:05.436651 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260810 19:24:05.929080 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:05.928224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:34917)
0810 19:24:05.928760 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:05.928908 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260810 19:24:05.974706 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:05.973976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:56207)
0810 19:24:05.974450 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:05.974535 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
I20260810 19:24:06.438737 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:06.437895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:34177)
0810 19:24:06.438401 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:06.438523 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
I20260810 19:24:06.931026 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:06.930149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:43097)
0810 19:24:06.930641 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:06.930736 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260810 19:24:06.976446 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:06.975680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:48041)
0810 19:24:06.976165 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:06.976253 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260810 19:24:07.440860 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:07.439831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:50409)
0810 19:24:07.440475 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:07.440602 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":306,"threads_started":1}
I20260810 19:24:07.932797 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:07.932042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:38559)
0810 19:24:07.932531 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:07.932621 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
I20260810 19:24:07.978233 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:07.977422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:60847)
0810 19:24:07.977940 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:07.978032 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
I20260810 19:24:08.442659 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:08.441902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:37225)
0810 19:24:08.442391 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:08.442483 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
I20260810 19:24:08.934950 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:08.934079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:40403)
0810 19:24:08.934601 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:08.934717 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":267,"threads_started":1}
I20260810 19:24:08.980197 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:08.979334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:56411)
0810 19:24:08.979889 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:08.979991 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":260,"threads_started":1}
I20260810 19:24:09.444327 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:09.443598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:46097)
0810 19:24:09.444070 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:09.444157 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":233,"threads_started":1}
I20260810 19:24:09.936834 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:09.936010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:41391)
0810 19:24:09.936510 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:09.936606 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":253,"threads_started":1}
I20260810 19:24:09.982095 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:09.981180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:57011)
0810 19:24:09.981742 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:09.981843 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":332,"threads_started":1}
I20260810 19:24:10.316288 3470 raft_consensus.cc:482] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:10.316502 3470 raft_consensus.cc:504] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc9fbe8df0847d985170952e5932353" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 41089 } } peers { permanent_uuid: "16bd5000dd1a488997d78ba43b14ae16" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 35729 } } peers { permanent_uuid: "7a1386f40f734df89b4135cf7a34b356" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35497 } }
I20260810 19:24:10.317226 3470 leader_election.cc:288] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729), 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497)
W20260810 19:24:10.318172 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd5000dd1a488997d78ba43b14ae16 (127.7.43.2:35729): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
W20260810 19:24:10.318360 2809 leader_election.cc:341] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a1386f40f734df89b4135cf7a34b356 (127.7.43.3:35497): Not found: Tablet not found: 828a1a37a74b4aadb0203100723cfcfa
I20260810 19:24:10.318449 2809 leader_election.cc:302] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dc9fbe8df0847d985170952e5932353; no voters: 16bd5000dd1a488997d78ba43b14ae16, 7a1386f40f734df89b4135cf7a34b356
I20260810 19:24:10.318691 3470 raft_consensus.cc:2726] T 828a1a37a74b4aadb0203100723cfcfa P 8dc9fbe8df0847d985170952e5932353 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:24:10.446113 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:10.445263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:52177)
0810 19:24:10.445770 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:10.445902 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260810 19:24:10.938835 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:10.937938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:38033)
0810 19:24:10.938449 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:10.938585 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":279,"threads_started":1}
I20260810 19:24:10.984295 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:10.983400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.1:54317)
0810 19:24:10.983966 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:10.984077 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":282,"threads_started":1}
I20260810 19:24:11.447932 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:11.447220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.2:55387)
0810 19:24:11.447660 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:11.447761 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":216,"threads_started":1}
I20260810 19:24:11.913236 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 2706
I20260810 19:24:11.934239 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:11.914438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.7.43.1:41089)
0810 19:24:11.914997 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:11.915005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:11.915035 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:24:11.915099 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:11.915107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:11.915327 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:11.915471 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:11.917010 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:11.918160 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:11.919530 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:11.920122 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:11.920224 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:11.928638 (+ 8414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:11.928653 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:11.928673 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:11.928714 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:11.931574 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:11.932305 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:11.932312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:11.932315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:11.932467 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:11.932763 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:11.932766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:11.932768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:11.933129 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:11.933402 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:11.933662 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 19:24:11.933888 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260810 19:24:11.935532 2960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:11.940701 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:11.939985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:36245 (local address 127.7.43.3:52585)
0810 19:24:11.940442 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:11.940525 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.62:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260810 19:24:12.026467 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 2706
I20260810 19:24:12.058143 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 2976
I20260810 19:24:12.074501 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:12.059050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.7.43.2:35729)
0810 19:24:12.059526 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:12.059530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:12.059546 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:24:12.059605 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:12.059609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:12.059673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:12.059776 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:12.061599 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:12.062697 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:12.064483 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:12.064909 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:12.064999 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:12.068496 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:12.068501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:12.068505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:12.068541 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:12.071235 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:12.072034 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:12.072037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:12.072040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:12.072177 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:12.072575 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:12.072578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:12.072580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:12.072839 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:12.073051 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:12.073830 (+ 779us) server_negotiation.cc:299] Negotiation successful
0810 19:24:12.074070 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
I20260810 19:24:12.075790 3175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:12.162256 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 2976
I20260810 19:24:12.186627 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 3188
I20260810 19:24:12.200631 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:12.187510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.7.43.3:35497)
0810 19:24:12.188083 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:12.188088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:12.188106 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:24:12.188163 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:12.188166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:12.188228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:12.188325 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:12.189862 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:12.190721 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:12.192022 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:12.192490 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:12.192616 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:12.195393 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:12.195398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:12.195402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:12.195437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:12.197970 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:12.198628 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:12.198633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:12.198635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:12.198766 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:12.199159 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:12.199163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:12.199165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:12.199463 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:12.199766 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:12.200041 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 19:24:12.200259 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260810 19:24:12.201411 3381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:12.284468 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 3188
2026-08-10T19:24:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64917 ms)
[----------] 30 tests from SecurityITest (310270 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:12 dist-test-slave-7lbf krb5kdc[3491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:12 dist-test-slave-7lbf krb5kdc[3491](info): set up 2 sockets
Aug 10 19:24:12 dist-test-slave-7lbf krb5kdc[3491](info): commencing operation
krb5kdc: starting...
W20260810 19:24:13.556020 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:13 dist-test-slave-7lbf krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:24:13.877178 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:45753
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:14.226145 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:14.226485 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:14.226641 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:14.234082 3511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:14.234222 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:14.234304 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:14.234383 3511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:14.234448 3511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:14.244614 3511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:45753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:14.246026 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:14.247377 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:14.259114 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:14.259155 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:14.259205 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:14.262898 3511 server_base.cc:1034] running on GCE node
I20260810 19:24:14.263672 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:14.265080 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:14.266312 3511 hybrid_clock.cc:648] HybridClock initialized: now 1786389854266253 us; error 37 us; skew 500 ppm
I20260810 19:24:14.266661 3511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:14 dist-test-slave-7lbf krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:14.269786 3511 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:24:14.271409 3511 webserver.cc:460] Webserver started at http://127.7.43.62:34351/ using document root <none> and password file <none>
I20260810 19:24:14.272101 3511 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:14.272203 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:14.272460 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:14.275521 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "1fee016f72014845b2bd4434b18e5e1c"
format_stamp: "Formatted at 2026-08-10 19:24:14 on dist-test-slave-7lbf"
server_key: "345aacee494b0be51cfd1b2d569fe929"
server_key_iv: "e47e500c1a0f3643c5ddc665fc28bfd0"
server_key_version: "encryptionkey@0"
I20260810 19:24:14.276302 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "1fee016f72014845b2bd4434b18e5e1c"
format_stamp: "Formatted at 2026-08-10 19:24:14 on dist-test-slave-7lbf"
server_key: "345aacee494b0be51cfd1b2d569fe929"
server_key_iv: "e47e500c1a0f3643c5ddc665fc28bfd0"
server_key_version: "encryptionkey@0"
I20260810 19:24:14.281595 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:24:14.284978 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:14.286358 3511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:14.286504 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "1fee016f72014845b2bd4434b18e5e1c"
format_stamp: "Formatted at 2026-08-10 19:24:14 on dist-test-slave-7lbf"
server_key: "345aacee494b0be51cfd1b2d569fe929"
server_key_iv: "e47e500c1a0f3643c5ddc665fc28bfd0"
server_key_version: "encryptionkey@0"
I20260810 19:24:14.286679 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:14.303025 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:14.306842 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:14.307093 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:14.325874 3511 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:45753
I20260810 19:24:14.325888 3603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:45753 every 8 connection(s)
I20260810 19:24:14.327286 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:14.331513 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:14.337281 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 3511
I20260810 19:24:14.337569 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:14.338557 7340 external_mini_cluster.cc:1428] Setting key 1e7086c4636121cf36d731077cb5c303
I20260810 19:24:14.344484 3607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c: Bootstrap starting.
I20260810 19:24:14.347973 3607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:14.349062 3607 log.cc:824] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:14.352222 3607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c: No bootstrap required, opened a new log
Aug 10 19:24:14 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:14.363113 3607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fee016f72014845b2bd4434b18e5e1c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45753 } }
I20260810 19:24:14.363641 3607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:14.363765 3607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fee016f72014845b2bd4434b18e5e1c, State: Initialized, Role: FOLLOWER
I20260810 19:24:14.364185 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:14.341185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.7.43.62:45753)
0810 19:24:14.342224 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:14.342236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:14.342289 (+ 53us) server_negotiation.cc:407] Connection header received
0810 19:24:14.344148 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:14.344172 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:14.344587 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:14.344989 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:14.346563 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:14.347879 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:14.349280 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:14.350276 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:14.350495 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:14.354951 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:14.354963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:14.354988 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:14.355056 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:14.359279 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:14.359999 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:14.360009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:14.360019 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:14.360231 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:14.360642 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:14.360648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:14.360651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:14.361455 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:14.361809 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:14.362184 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 19:24:14.362650 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":534,"threads_started":1}
I20260810 19:24:14.364511 3607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fee016f72014845b2bd4434b18e5e1c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45753 } }
I20260810 19:24:14.364787 3607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:14.364964 3607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:14.365157 3607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:14.367674 3607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fee016f72014845b2bd4434b18e5e1c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45753 } }
I20260810 19:24:14.368201 3607 leader_election.cc:302] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fee016f72014845b2bd4434b18e5e1c; no voters:
I20260810 19:24:14.368809 3607 leader_election.cc:288] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:14.368932 3618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:14.369711 3618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [term 1 LEADER]: Becoming Leader. State: Replica: 1fee016f72014845b2bd4434b18e5e1c, State: Running, Role: LEADER
I20260810 19:24:14.370244 3618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fee016f72014845b2bd4434b18e5e1c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45753 } }
I20260810 19:24:14.371042 3607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:14.373061 3619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fee016f72014845b2bd4434b18e5e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fee016f72014845b2bd4434b18e5e1c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45753 } } }
I20260810 19:24:14.373553 3619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:14.375805 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fee016f72014845b2bd4434b18e5e1c. Latest consensus state: current_term: 1 leader_uuid: "1fee016f72014845b2bd4434b18e5e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fee016f72014845b2bd4434b18e5e1c" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 45753 } } }
I20260810 19:24:14.376165 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:14.378633 3629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:14.384291 3629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:14.394140 3629 catalog_manager.cc:1347] Generated new cluster ID: 582a31dd94db481486c41a9ac5c0542f
I20260810 19:24:14.394330 3629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:14.405897 3629 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:14.407202 3629 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:14.432719 3629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c: Generated new TSK 0
I20260810 19:24:14.433842 3629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:24:14.527074 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45753
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:14.897045 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:14.897343 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:14.897439 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:14.904665 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:14.904779 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:14.904908 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:24:14.914731 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:14.916004 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:14.917299 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:14.930557 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:14.930521 3661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:14.930613 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:14.931543 3646 server_base.cc:1034] running on GCE node
I20260810 19:24:14.932243 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:14.933274 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:14.934484 3646 hybrid_clock.cc:648] HybridClock initialized: now 1786389854934422 us; error 45 us; skew 500 ppm
I20260810 19:24:14.934775 3646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:14 dist-test-slave-7lbf krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:14.938019 3646 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:24:14.939427 3646 webserver.cc:460] Webserver started at http://127.7.43.1:45183/ using document root <none> and password file <none>
I20260810 19:24:14.940124 3646 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:14.940238 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:14.940549 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:14.943645 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "334df8dd5246447a88db2e0289f994b6"
format_stamp: "Formatted at 2026-08-10 19:24:14 on dist-test-slave-7lbf"
server_key: "4fed5b5a6ffaba4d0012c887d026967e"
server_key_iv: "475bb7477824bb6ef91aa872c19738f7"
server_key_version: "encryptionkey@0"
I20260810 19:24:14.944417 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "334df8dd5246447a88db2e0289f994b6"
format_stamp: "Formatted at 2026-08-10 19:24:14 on dist-test-slave-7lbf"
server_key: "4fed5b5a6ffaba4d0012c887d026967e"
server_key_iv: "475bb7477824bb6ef91aa872c19738f7"
server_key_version: "encryptionkey@0"
I20260810 19:24:14.949651 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:24:14.953090 3668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:14.954407 3646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:14.954551 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "334df8dd5246447a88db2e0289f994b6"
format_stamp: "Formatted at 2026-08-10 19:24:14 on dist-test-slave-7lbf"
server_key: "4fed5b5a6ffaba4d0012c887d026967e"
server_key_iv: "475bb7477824bb6ef91aa872c19738f7"
server_key_version: "encryptionkey@0"
I20260810 19:24:14.954735 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:14.966866 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:14.971190 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:14.971484 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:14.972517 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:14.973932 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:14.974007 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:14.974123 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:14.974175 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:15.007354 3646 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:34147
I20260810 19:24:15.007390 3846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:34147 every 8 connection(s)
I20260810 19:24:15.008773 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:24:15.013909 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 3646
I20260810 19:24:15.014113 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:24:15.014768 7340 external_mini_cluster.cc:1428] Setting key 65c7717045d090672a38e2adfa0cbc54
Aug 10 19:24:15 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389854, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:15.031843 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:15.011909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45753 (local address 127.7.43.1:54065)
0810 19:24:15.012705 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:15.012755 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:15.014699 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:15.015462 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:15.015472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:15.016152 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:15.016872 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:15.016896 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.018302 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.018306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:15.019626 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:15.019640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.020539 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.020544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:15.020765 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:15.021459 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:15.021483 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:15.023916 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:15.027440 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:15.027447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:15.027461 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:15.027907 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:15.028314 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:15.028318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:15.028321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:15.028562 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:15.029170 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:15.029177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:15.029594 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 19:24:15.029858 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 19:24:15.030333 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":298,"threads_started":1}
I20260810 19:24:15.031877 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:15.011450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:54065 (local address 127.7.43.62:45753)
0810 19:24:15.011974 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:15.011980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:15.012893 (+ 913us) server_negotiation.cc:407] Connection header received
0810 19:24:15.015004 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:15.015010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:15.015103 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:15.015229 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:15.017053 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.018112 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.019775 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.020294 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.020419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:15.024141 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:15.024147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:15.024151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:15.024190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:15.027294 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:15.028035 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:15.028039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:15.028042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:15.028190 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:15.028678 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:15.028683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:15.028685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:15.028986 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:15.029106 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:15.031279 (+ 2173us) server_negotiation.cc:299] Negotiation successful
0810 19:24:15.031515 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260810 19:24:15.033427 3847 heartbeater.cc:347] Connected to a master server at 127.7.43.62:45753
I20260810 19:24:15.033805 3847 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:15.034735 3847 heartbeater.cc:511] Master 127.7.43.62:45753 requested a full tablet report, sending...
I20260810 19:24:15.037050 3558 ts_manager.cc:194] Registered new tserver with Master: 334df8dd5246447a88db2e0289f994b6 (127.7.43.1:34147)
I20260810 19:24:15.038591 3558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:54065
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:24:15.084367 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45753
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:15.430523 3856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:15.430847 3856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:15.430958 3856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:15.438378 3856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:15.438501 3856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:15.438625 3856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:24:15.448385 3856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:15.449819 3856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:15.451162 3856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:15.464622 3863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:15.464645 3864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:15.464766 3869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:15.465592 3856 server_base.cc:1034] running on GCE node
I20260810 19:24:15.466243 3856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:15.467242 3856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:15.468447 3856 hybrid_clock.cc:648] HybridClock initialized: now 1786389855468392 us; error 49 us; skew 500 ppm
I20260810 19:24:15.468772 3856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:15 dist-test-slave-7lbf krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:15.471938 3856 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:24:15.473433 3856 webserver.cc:460] Webserver started at http://127.7.43.2:33635/ using document root <none> and password file <none>
I20260810 19:24:15.474159 3856 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:15.474265 3856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:15.474536 3856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:15.477617 3856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "ab292ba5d003462f897909a1005f5de9"
format_stamp: "Formatted at 2026-08-10 19:24:15 on dist-test-slave-7lbf"
server_key: "b1018b1588cbc186b97caaaeb22c36d3"
server_key_iv: "152d946a6707cfe20632cde1a4698b6f"
server_key_version: "encryptionkey@0"
I20260810 19:24:15.478483 3856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "ab292ba5d003462f897909a1005f5de9"
format_stamp: "Formatted at 2026-08-10 19:24:15 on dist-test-slave-7lbf"
server_key: "b1018b1588cbc186b97caaaeb22c36d3"
server_key_iv: "152d946a6707cfe20632cde1a4698b6f"
server_key_version: "encryptionkey@0"
I20260810 19:24:15.483707 3856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:24:15.487177 3880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:15.488507 3856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:24:15.488655 3856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "ab292ba5d003462f897909a1005f5de9"
format_stamp: "Formatted at 2026-08-10 19:24:15 on dist-test-slave-7lbf"
server_key: "b1018b1588cbc186b97caaaeb22c36d3"
server_key_iv: "152d946a6707cfe20632cde1a4698b6f"
server_key_version: "encryptionkey@0"
I20260810 19:24:15.488842 3856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:15.509696 3856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:15.513689 3856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:15.513942 3856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:15.514915 3856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:15.516217 3856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:15.516297 3856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:15.516387 3856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:15.516431 3856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:15.547696 3856 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:40947
I20260810 19:24:15.547742 4080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:40947 every 8 connection(s)
I20260810 19:24:15.549012 3856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:24:15.556432 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 3856
I20260810 19:24:15.556623 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:24:15.557217 7340 external_mini_cluster.cc:1428] Setting key 9b2ba13fa2e1ebac9356808498061cf9
Aug 10 19:24:15 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389855, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:15.570262 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:15.551654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:47957 (local address 127.7.43.62:45753)
0810 19:24:15.552086 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:15.552091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:15.553148 (+ 1057us) server_negotiation.cc:407] Connection header received
0810 19:24:15.555066 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:15.555071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:15.555142 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:15.555237 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:15.557018 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.557792 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.559815 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.560441 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.560577 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:15.564202 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:15.564208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:15.564213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:15.564250 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:15.567258 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:15.568011 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:15.568016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:15.568019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:15.568180 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:15.568583 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:15.568587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:15.568589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:15.568893 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:15.569025 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:15.569684 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 19:24:15.569894 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260810 19:24:15.571174 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:15.552213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45753 (local address 127.7.43.2:47957)
0810 19:24:15.552972 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:15.553023 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:15.554785 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:15.555466 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:15.555474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:15.555991 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:15.556826 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:15.556854 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.557956 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.557961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:15.559671 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:15.559684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:15.560669 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:15.560675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:15.560880 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:15.561668 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:15.561693 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:15.563972 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:15.567404 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:15.567411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:15.567424 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:15.567833 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:15.568291 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:15.568294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:15.568296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:15.568484 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:15.569000 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:15.569005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:15.569306 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 19:24:15.569570 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 19:24:15.569979 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":292,"threads_started":1}
I20260810 19:24:15.572623 4082 heartbeater.cc:347] Connected to a master server at 127.7.43.62:45753
I20260810 19:24:15.572921 4082 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:15.573797 4082 heartbeater.cc:511] Master 127.7.43.62:45753 requested a full tablet report, sending...
I20260810 19:24:15.575398 3558 ts_manager.cc:194] Registered new tserver with Master: ab292ba5d003462f897909a1005f5de9 (127.7.43.2:40947)
I20260810 19:24:15.576177 3558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:47957
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:24:15.628597 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45753
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:15.982575 4088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:15.982949 4088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:15.983114 4088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:15.990716 4088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:15.990870 4088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:15.991042 4088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:24:16.001235 4088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:16.002570 4088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:16.003954 4088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:16.017585 4095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:16.017644 4099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:16.017671 4093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:16.018458 4088 server_base.cc:1034] running on GCE node
I20260810 19:24:16.019075 4088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:16.020020 4088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:16.021231 4088 hybrid_clock.cc:648] HybridClock initialized: now 1786389856021175 us; error 46 us; skew 500 ppm
I20260810 19:24:16.021596 4088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:16 dist-test-slave-7lbf krb5kdc[3491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389856, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:16.024729 4088 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:24:16.026286 4088 webserver.cc:460] Webserver started at http://127.7.43.3:32989/ using document root <none> and password file <none>
I20260810 19:24:16.026994 4088 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:16.027115 4088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:16.027423 4088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:16.030498 4088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "84fb9a020e454844a07ab08b60909418"
format_stamp: "Formatted at 2026-08-10 19:24:16 on dist-test-slave-7lbf"
server_key: "d77a9db8f9e871d205ccba7f7f42e848"
server_key_iv: "cdc46c701cd416b997dc836c8fcf0c0e"
server_key_version: "encryptionkey@0"
I20260810 19:24:16.031296 4088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "84fb9a020e454844a07ab08b60909418"
format_stamp: "Formatted at 2026-08-10 19:24:16 on dist-test-slave-7lbf"
server_key: "d77a9db8f9e871d205ccba7f7f42e848"
server_key_iv: "cdc46c701cd416b997dc836c8fcf0c0e"
server_key_version: "encryptionkey@0"
I20260810 19:24:16.036547 4088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 19:24:16.040155 4113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:16.041668 4088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:16.041838 4088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "84fb9a020e454844a07ab08b60909418"
format_stamp: "Formatted at 2026-08-10 19:24:16 on dist-test-slave-7lbf"
server_key: "d77a9db8f9e871d205ccba7f7f42e848"
server_key_iv: "cdc46c701cd416b997dc836c8fcf0c0e"
server_key_version: "encryptionkey@0"
I20260810 19:24:16.042096 3847 heartbeater.cc:503] Master 127.7.43.62:45753 was elected leader, sending a full tablet report...
I20260810 19:24:16.042193 4088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:16.062003 4088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:16.066011 4088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:16.066272 4088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:16.067241 4088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:16.068609 4088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:16.068681 4088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:16.068795 4088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:16.068843 4088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:16.100831 4088 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:46251
I20260810 19:24:16.100880 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:46251 every 8 connection(s)
I20260810 19:24:16.102221 4088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:24:16.110200 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 4088
I20260810 19:24:16.110404 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:24:16.110877 7340 external_mini_cluster.cc:1428] Setting key fd50b792d3c25bf82fe690555568c262
Aug 10 19:24:16 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389856, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:16.122475 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.104749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:45031 (local address 127.7.43.62:45753)
0810 19:24:16.105183 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.105187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.106096 (+ 909us) server_negotiation.cc:407] Connection header received
0810 19:24:16.107921 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.107927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.108010 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.108108 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:16.109644 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.110476 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.112357 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.112941 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.113070 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.116902 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:16.116909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:16.116914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:16.116964 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:16.119658 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.120363 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.120367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.120369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.120513 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.120874 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.120877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.120879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.121160 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:16.121271 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.122013 (+ 742us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.122189 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":223,"threads_started":1}
I20260810 19:24:16.123449 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.105172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:45753 (local address 127.7.43.3:45031)
0810 19:24:16.105924 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:16.105972 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:16.107650 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:16.108359 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:16.108367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:16.108871 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:16.109451 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.109469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.110639 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.110643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.112188 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.112204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.113208 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.113217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.113558 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.114379 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:16.114409 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:16.116667 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:16.119779 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:16.119786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:16.119801 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:16.120267 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:16.120591 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:16.120594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:16.120596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:16.120784 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:16.121253 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:16.121259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:16.121632 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 19:24:16.121871 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 19:24:16.122298 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":303,"threads_started":1}
I20260810 19:24:16.125007 4299 heartbeater.cc:347] Connected to a master server at 127.7.43.62:45753
I20260810 19:24:16.125335 4299 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:16.126264 4299 heartbeater.cc:511] Master 127.7.43.62:45753 requested a full tablet report, sending...
I20260810 19:24:16.127843 3558 ts_manager.cc:194] Registered new tserver with Master: 84fb9a020e454844a07ab08b60909418 (127.7.43.3:46251)
I20260810 19:24:16.128871 3558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:45031
I20260810 19:24:16.139645 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:24:16.157625 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.144429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.7.43.62:45753)
0810 19:24:16.144654 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.144659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.145095 (+ 436us) server_negotiation.cc:407] Connection header received
0810 19:24:16.145245 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.145249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.145319 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.145424 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:16.146983 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.147751 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.149038 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.149471 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.149623 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.152131 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:16.152137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:16.152140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:16.152179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:16.154881 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.155621 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.155624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.155627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.155764 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.156061 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.156064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.156066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.156419 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:16.156619 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.157020 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.157251 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 19:24:16.162267 3558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:24:16.169324 3558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:24:16.193688 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.182730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.7.43.1:34147)
0810 19:24:16.183429 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.183439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.184712 (+ 1273us) server_negotiation.cc:407] Connection header received
0810 19:24:16.184835 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.184844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.185118 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.186183 (+ 1065us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:16.186426 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.188119 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.191228 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.192319 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.192453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.192720 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.193155 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.193331 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":323,"threads_started":1}
I20260810 19:24:16.193747 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.183142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.7.43.2:40947)
0810 19:24:16.184600 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.184608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.184638 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:24:16.184894 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.184904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.185156 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.185374 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:16.186412 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.188087 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.191235 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.192684 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.192797 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.192986 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.193242 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.193418 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":1268,"threads_started":1}
I20260810 19:24:16.193892 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.182714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:40947 (local address 127.0.0.1:37164)
0810 19:24:16.184478 (+ 1764us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:16.184501 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:16.184613 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:16.185655 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:16.185661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:16.185691 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:16.186195 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.186209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.188289 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.188295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.191032 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.191045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.192905 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.192911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.193064 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.193095 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:24:16.193186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 19:24:16.193388 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"mutex_wait_us":48,"thread_start_us":721,"threads_started":1}
I20260810 19:24:16.194945 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.182693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34147 (local address 127.0.0.1:56790)
0810 19:24:16.183848 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:16.183894 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:16.184102 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:16.185371 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:16.185379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:16.185414 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:16.185923 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.185937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.188289 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.188296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.191035 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.191051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.192496 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.192503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.192697 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.192751 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 19:24:16.194337 (+ 1586us) client_negotiation.cc:240] Negotiation successful
0810 19:24:16.194480 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":458,"thread_start_us":1464,"threads_started":1}
I20260810 19:24:16.197832 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.184585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:46251 (local address 127.0.0.1:52714)
0810 19:24:16.189420 (+ 4835us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:16.189442 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:16.189601 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:16.190507 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:16.190512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:16.190562 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:16.191064 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.191075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.193082 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.193087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.195548 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.195562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.197004 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.197011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.197174 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.197207 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:16.197303 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 19:24:16.197431 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4707,"mutex_wait_us":47,"thread_start_us":4549,"threads_started":1}
I20260810 19:24:16.197921 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.185036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.7.43.3:46251)
0810 19:24:16.188915 (+ 3879us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.188924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.189587 (+ 663us) server_negotiation.cc:407] Connection header received
0810 19:24:16.189918 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.189926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.190224 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.191065 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:16.191359 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.192923 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.195722 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.196784 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.196940 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.197180 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.197370 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.197582 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3706,"thread_start_us":3541,"threads_started":1}
I20260810 19:24:16.198132 3969 tablet_service.cc:1467] Processing CreateTablet for tablet 685305e651a04adc840ba34fe8019773 (DEFAULT_TABLE table=test-table [id=ea647c8efbb84e52a5c9842ec9f2b486]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:16.199522 3969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 685305e651a04adc840ba34fe8019773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:16.199277 3747 tablet_service.cc:1467] Processing CreateTablet for tablet 685305e651a04adc840ba34fe8019773 (DEFAULT_TABLE table=test-table [id=ea647c8efbb84e52a5c9842ec9f2b486]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:16.200820 3747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 685305e651a04adc840ba34fe8019773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:16.202656 4192 tablet_service.cc:1467] Processing CreateTablet for tablet 685305e651a04adc840ba34fe8019773 (DEFAULT_TABLE table=test-table [id=ea647c8efbb84e52a5c9842ec9f2b486]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:16.204028 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 685305e651a04adc840ba34fe8019773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:16.214570 4330 tablet_bootstrap.cc:492] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Bootstrap starting.
I20260810 19:24:16.216786 4331 tablet_bootstrap.cc:492] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Bootstrap starting.
I20260810 19:24:16.218593 4330 tablet_bootstrap.cc:654] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:16.218920 4333 tablet_bootstrap.cc:492] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Bootstrap starting.
I20260810 19:24:16.220044 4330 log.cc:824] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:16.220728 4331 tablet_bootstrap.cc:654] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:16.221922 4331 log.cc:824] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:16.222187 4333 tablet_bootstrap.cc:654] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:16.223450 4330 tablet_bootstrap.cc:492] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: No bootstrap required, opened a new log
I20260810 19:24:16.223477 4333 log.cc:824] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:16.223716 4330 ts_tablet_manager.cc:1397] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:24:16.225657 4331 tablet_bootstrap.cc:492] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: No bootstrap required, opened a new log
I20260810 19:24:16.225926 4331 ts_tablet_manager.cc:1397] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 19:24:16.225999 4333 tablet_bootstrap.cc:492] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: No bootstrap required, opened a new log
I20260810 19:24:16.226248 4333 ts_tablet_manager.cc:1397] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260810 19:24:16.233656 4333 raft_consensus.cc:348] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.234179 4333 raft_consensus.cc:374] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:16.234323 4333 raft_consensus.cc:729] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fb9a020e454844a07ab08b60909418, State: Initialized, Role: FOLLOWER
I20260810 19:24:16.234408 4330 raft_consensus.cc:348] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.234891 4330 raft_consensus.cc:374] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:16.235003 4330 raft_consensus.cc:729] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 334df8dd5246447a88db2e0289f994b6, State: Initialized, Role: FOLLOWER
I20260810 19:24:16.235025 4333 consensus_queue.cc:260] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.235746 4330 consensus_queue.cc:260] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.236181 4299 heartbeater.cc:503] Master 127.7.43.62:45753 was elected leader, sending a full tablet report...
I20260810 19:24:16.236372 4331 raft_consensus.cc:348] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.236853 4331 raft_consensus.cc:374] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:16.236940 4331 raft_consensus.cc:729] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab292ba5d003462f897909a1005f5de9, State: Initialized, Role: FOLLOWER
I20260810 19:24:16.236946 4333 ts_tablet_manager.cc:1428] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 19:24:16.237720 4331 consensus_queue.cc:260] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.238879 4330 ts_tablet_manager.cc:1428] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260810 19:24:16.239987 4082 heartbeater.cc:503] Master 127.7.43.62:45753 was elected leader, sending a full tablet report...
I20260810 19:24:16.240600 4331 ts_tablet_manager.cc:1428] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260810 19:24:16.261236 3848 tablet.cc:2369] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:24:16.301040 4083 tablet.cc:2369] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:24:16.353906 4301 tablet.cc:2369] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:24:16.426827 4339 raft_consensus.cc:482] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:16.427143 4339 raft_consensus.cc:504] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.429003 4339 leader_election.cc:288] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 334df8dd5246447a88db2e0289f994b6 (127.7.43.1:34147), ab292ba5d003462f897909a1005f5de9 (127.7.43.2:40947)
I20260810 19:24:16.435539 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.429571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:36183 (local address 127.7.43.1:34147)
0810 19:24:16.429788 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.429795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.429958 (+ 163us) server_negotiation.cc:407] Connection header received
0810 19:24:16.430149 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.430154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.430215 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.430293 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:16.430725 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.431692 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.433508 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.434669 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.434768 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.434847 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.435220 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.435336 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:24:16.435588 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.429449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:34147 (local address 127.7.43.3:36183)
0810 19:24:16.429838 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:16.429859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:16.430011 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:16.430334 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:16.430339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:16.430367 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:16.430574 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.430585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.431836 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.431842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.433340 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.433353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.434824 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.434830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.435036 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.435073 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 19:24:16.435173 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:24:16.435308 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260810 19:24:16.436414 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.429708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:34663 (local address 127.7.43.2:40947)
0810 19:24:16.429891 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.429896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.430513 (+ 617us) server_negotiation.cc:407] Connection header received
0810 19:24:16.430664 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.430670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.430743 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.430809 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:16.431432 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.432351 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.434411 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.435437 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.435579 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.435668 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.436085 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.436212 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:24:16.436416 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.429491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:40947 (local address 127.7.43.3:34663)
0810 19:24:16.430392 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:16.430414 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:16.430528 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:16.430870 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:16.430874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:16.430899 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:16.431263 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.431276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.432520 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.432524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.434256 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:16.434271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.435701 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.435708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:16.435929 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.435956 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:24:16.436033 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 19:24:16.436127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":532,"threads_started":1}
I20260810 19:24:16.436878 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685305e651a04adc840ba34fe8019773" candidate_uuid: "84fb9a020e454844a07ab08b60909418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "334df8dd5246447a88db2e0289f994b6" is_pre_election: true
I20260810 19:24:16.437306 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685305e651a04adc840ba34fe8019773" candidate_uuid: "84fb9a020e454844a07ab08b60909418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab292ba5d003462f897909a1005f5de9" is_pre_election: true
I20260810 19:24:16.437420 3778 raft_consensus.cc:2454] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84fb9a020e454844a07ab08b60909418 in term 0.
I20260810 19:24:16.437788 4000 raft_consensus.cc:2454] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84fb9a020e454844a07ab08b60909418 in term 0.
I20260810 19:24:16.438242 4119 leader_election.cc:302] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334df8dd5246447a88db2e0289f994b6, 84fb9a020e454844a07ab08b60909418; no voters:
I20260810 19:24:16.438638 4339 raft_consensus.cc:2781] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:24:16.438768 4339 raft_consensus.cc:482] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:24:16.438872 4339 raft_consensus.cc:3037] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:16.441170 4339 raft_consensus.cc:504] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.441905 4339 leader_election.cc:288] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [CANDIDATE]: Term 1 election: Requested vote from peers 334df8dd5246447a88db2e0289f994b6 (127.7.43.1:34147), ab292ba5d003462f897909a1005f5de9 (127.7.43.2:40947)
I20260810 19:24:16.442438 4000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685305e651a04adc840ba34fe8019773" candidate_uuid: "84fb9a020e454844a07ab08b60909418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab292ba5d003462f897909a1005f5de9"
I20260810 19:24:16.442440 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685305e651a04adc840ba34fe8019773" candidate_uuid: "84fb9a020e454844a07ab08b60909418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "334df8dd5246447a88db2e0289f994b6"
I20260810 19:24:16.442662 4000 raft_consensus.cc:3037] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:16.442662 3778 raft_consensus.cc:3037] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:16.445883 4000 raft_consensus.cc:2454] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84fb9a020e454844a07ab08b60909418 in term 1.
I20260810 19:24:16.445881 3778 raft_consensus.cc:2454] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84fb9a020e454844a07ab08b60909418 in term 1.
I20260810 19:24:16.446369 4120 leader_election.cc:302] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 334df8dd5246447a88db2e0289f994b6, 84fb9a020e454844a07ab08b60909418, ab292ba5d003462f897909a1005f5de9; no voters:
I20260810 19:24:16.446715 4339 raft_consensus.cc:2781] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:16.447177 4339 raft_consensus.cc:686] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 1 LEADER]: Becoming Leader. State: Replica: 84fb9a020e454844a07ab08b60909418, State: Running, Role: LEADER
I20260810 19:24:16.447739 4339 consensus_queue.cc:237] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } }
I20260810 19:24:16.452461 3556 catalog_manager.cc:5495] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84fb9a020e454844a07ab08b60909418 (127.7.43.3). New cstate: current_term: 1 leader_uuid: "84fb9a020e454844a07ab08b60909418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84fb9a020e454844a07ab08b60909418" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 46251 } health_report { overall_health: HEALTHY } } }
I20260810 19:24:16.499933 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.493666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.7.43.3:46251)
0810 19:24:16.493860 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.493866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.493903 (+ 37us) server_negotiation.cc:407] Connection header received
0810 19:24:16.494136 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.494142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.494268 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.494448 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:24:16.494770 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.495882 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.497277 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.497973 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.498101 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.498601 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:24:16.499329 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:24:16.499504 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.499553 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.499642 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:24:16.509626 3778 raft_consensus.cc:1261] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 1 FOLLOWER]: Refusing update from remote peer 84fb9a020e454844a07ab08b60909418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:16.509626 4000 raft_consensus.cc:1261] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 1 FOLLOWER]: Refusing update from remote peer 84fb9a020e454844a07ab08b60909418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:16.510756 4346 consensus_queue.cc:1035] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334df8dd5246447a88db2e0289f994b6" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:16.511143 4339 consensus_queue.cc:1035] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab292ba5d003462f897909a1005f5de9" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:16.529760 4350 mvcc.cc:205] Tried to move back new op lower bound from 7317052852252254208 to 7317052852013727744. Current Snapshot: MvccSnapshot[applied={T|T < 7317052852252254208}]
I20260810 19:24:16.554840 3556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:16.555163 3556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:16.559715 3556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fee016f72014845b2bd4434b18e5e1c: Sending DeleteTablet for 3 replicas of tablet 685305e651a04adc840ba34fe8019773
I20260810 19:24:16.560943 3747 tablet_service.cc:1514] Processing DeleteTablet for tablet 685305e651a04adc840ba34fe8019773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:16 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:56790
I20260810 19:24:16.561235 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet 685305e651a04adc840ba34fe8019773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:16 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:52714
I20260810 19:24:16.562002 3969 tablet_service.cc:1514] Processing DeleteTablet for tablet 685305e651a04adc840ba34fe8019773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:16 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:37164
I20260810 19:24:16.564422 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 3646
I20260810 19:24:16.579417 4373 tablet_replica.cc:330] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: stopping tablet replica
I20260810 19:24:16.580829 4373 raft_consensus.cc:2229] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:16.581604 4368 tablet_replica.cc:330] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: stopping tablet replica
I20260810 19:24:16.582093 4373 raft_consensus.cc:2258] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:16.582154 4368 raft_consensus.cc:2229] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:16.582751 4368 raft_consensus.cc:2258] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:16.587968 4373 ts_tablet_manager.cc:1905] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:16.594540 4368 ts_tablet_manager.cc:1905] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:16.594631 4373 ts_tablet_manager.cc:1918] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:16.594813 4373 log.cc:1196] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/685305e651a04adc840ba34fe8019773
I20260810 19:24:16.595314 4373 ts_tablet_manager.cc:1939] T 685305e651a04adc840ba34fe8019773 P ab292ba5d003462f897909a1005f5de9: Deleting consensus metadata
I20260810 19:24:16.597060 3540 catalog_manager.cc:4841] TS ab292ba5d003462f897909a1005f5de9 (127.7.43.2:40947): tablet 685305e651a04adc840ba34fe8019773 (table test-table [id=ea647c8efbb84e52a5c9842ec9f2b486]) successfully deleted
I20260810 19:24:16.598342 4371 tablet_replica.cc:330] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: stopping tablet replica
I20260810 19:24:16.598860 4371 raft_consensus.cc:2229] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:24:16.599542 4371 raft_consensus.cc:2258] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:16.602128 4371 ts_tablet_manager.cc:1905] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:24:16 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:24:16.605640 4368 ts_tablet_manager.cc:1918] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:16.605862 4368 log.cc:1196] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/685305e651a04adc840ba34fe8019773
I20260810 19:24:16.606489 4368 ts_tablet_manager.cc:1939] T 685305e651a04adc840ba34fe8019773 P 334df8dd5246447a88db2e0289f994b6: Deleting consensus metadata
I20260810 19:24:16.608723 3539 catalog_manager.cc:4841] TS 334df8dd5246447a88db2e0289f994b6 (127.7.43.1:34147): tablet 685305e651a04adc840ba34fe8019773 (table test-table [id=ea647c8efbb84e52a5c9842ec9f2b486]) successfully deleted
I20260810 19:24:16.611272 4371 ts_tablet_manager.cc:1918] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:16.611590 4371 log.cc:1196] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/685305e651a04adc840ba34fe8019773
I20260810 19:24:16.612280 4371 ts_tablet_manager.cc:1939] T 685305e651a04adc840ba34fe8019773 P 84fb9a020e454844a07ab08b60909418: Deleting consensus metadata
I20260810 19:24:16.614723 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.566051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.7.43.1:34147)
0810 19:24:16.566236 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.566243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.576848 (+ 10605us) server_negotiation.cc:407] Connection header received
0810 19:24:16.577244 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.577251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.577360 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.577441 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:16.582357 (+ 4916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.583529 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.585571 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.586795 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.587021 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.604658 (+ 17637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:16.604679 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:16.604698 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:16.604758 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:16.609331 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.610786 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.610797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.610810 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.611102 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.611920 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.611925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.611929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.612695 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:16.613122 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.614034 (+ 912us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.614318 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:24:16.616824 3536 catalog_manager.cc:4841] TS 84fb9a020e454844a07ab08b60909418 (127.7.43.3:46251): tablet 685305e651a04adc840ba34fe8019773 (table test-table [id=ea647c8efbb84e52a5c9842ec9f2b486]) successfully deleted
I20260810 19:24:16.617009 3840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:16.783880 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 3646
I20260810 19:24:16.829238 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 3856
Aug 10 19:24:16 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:24:16.845221 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:16.830055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.7.43.2:40947)
0810 19:24:16.830317 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:16.830323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:16.830344 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:24:16.830474 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:16.830480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:16.830573 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:16.830658 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:16.831984 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.832744 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.834085 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:16.834508 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:16.834619 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:16.839011 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:16.839028 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:16.839045 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:16.839085 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:16.842087 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.842983 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.842991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.842993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.843137 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:16.843559 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:16.843564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:16.843566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:16.843988 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:16.844275 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:16.844618 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 19:24:16.844839 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 19:24:16.846076 4074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:16.984979 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 3856
I20260810 19:24:17.006606 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 4088
Aug 10 19:24:17 dist-test-slave-7lbf krb5kdc[3491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:24:17.028568 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:17.007427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.7.43.3:46251)
0810 19:24:17.008097 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:17.008104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:17.008136 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:24:17.008206 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:17.008212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:17.008312 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:17.008443 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:17.009869 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:17.011176 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:17.012579 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:17.013344 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:17.013509 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:17.021105 (+ 7596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:17.021130 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:17.021155 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:17.021210 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:17.025352 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:17.026196 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:17.026211 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:17.026214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:17.026395 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:17.026783 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:17.026788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:17.026791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:17.027363 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:17.027726 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:17.027920 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 19:24:17.028161 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":323,"threads_started":1}
I20260810 19:24:17.029418 4286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:17.153770 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 4088
I20260810 19:24:17.180665 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 3511
I20260810 19:24:17.181919 3599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:17.291785 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 3511
2026-08-10T19:24:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5019 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T19:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:17Z Disabled control of system clock
I20260810 19:24:17.397904 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42169
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:24:17.747504 4389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:17.747861 4389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:17.747963 4389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:17.755241 4389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:17.755353 4389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:17.755390 4389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:17.755448 4389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:17.755491 4389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:17.764902 4389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:42169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:17.766294 4389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:17.768162 4389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:17.780267 4398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:17.780256 4400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:17.780234 4403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:17.781170 4389 server_base.cc:1034] running on GCE node
I20260810 19:24:17.782055 4389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:17.783558 4389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:17.784770 4389 hybrid_clock.cc:648] HybridClock initialized: now 1786389857784722 us; error 37 us; skew 500 ppm
I20260810 19:24:17.785140 4389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:17.788390 4389 webserver.cc:460] Webserver started at http://127.7.43.62:32993/ using document root <none> and password file <none>
I20260810 19:24:17.789500 4389 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:17.789662 4389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:17.790074 4389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:17.794781 4389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "b99328da9e1c49bf89b0a8a31225ed31"
format_stamp: "Formatted at 2026-08-10 19:24:17 on dist-test-slave-7lbf"
server_key: "113c3528281745a2449768be79e41c30"
server_key_iv: "011029a2e625d1efaf873be6ea3f4ec7"
server_key_version: "encryptionkey@0"
I20260810 19:24:17.796159 4389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "b99328da9e1c49bf89b0a8a31225ed31"
format_stamp: "Formatted at 2026-08-10 19:24:17 on dist-test-slave-7lbf"
server_key: "113c3528281745a2449768be79e41c30"
server_key_iv: "011029a2e625d1efaf873be6ea3f4ec7"
server_key_version: "encryptionkey@0"
I20260810 19:24:17.803885 4389 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260810 19:24:17.809635 4415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:17.811647 4389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:24:17.811875 4389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "b99328da9e1c49bf89b0a8a31225ed31"
format_stamp: "Formatted at 2026-08-10 19:24:17 on dist-test-slave-7lbf"
server_key: "113c3528281745a2449768be79e41c30"
server_key_iv: "011029a2e625d1efaf873be6ea3f4ec7"
server_key_version: "encryptionkey@0"
I20260810 19:24:17.812170 4389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:17.847507 4389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:17.848443 4389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:17.848737 4389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:17.866957 4389 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:42169
I20260810 19:24:17.866978 4506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:42169 every 8 connection(s)
I20260810 19:24:17.868413 4389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:17.870954 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 4389
I20260810 19:24:17.871169 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:17.871835 7340 external_mini_cluster.cc:1428] Setting key 3b161f02023d6f886ebd429453ce361a
I20260810 19:24:17.873113 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:17.886497 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31: Bootstrap starting.
I20260810 19:24:17.887485 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:17.873954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.7.43.62:42169)
0810 19:24:17.876645 (+ 2691us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:17.876660 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:17.876708 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:24:17.878377 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:17.878403 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:17.878919 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:17.879334 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:17.879975 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:17.881320 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:17.882824 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:17.883582 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:17.883806 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:17.884976 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:17.885015 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:17.885032 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:17.885655 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:17.885743 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:17.885766 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:17.885870 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:17.886120 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 19:24:17.886447 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205,"thread_start_us":1979,"threads_started":1}
I20260810 19:24:17.890216 4507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:17.891427 4507 log.cc:824] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:17.894238 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31: No bootstrap required, opened a new log
I20260810 19:24:17.902046 4507 raft_consensus.cc:348] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99328da9e1c49bf89b0a8a31225ed31" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42169 } }
I20260810 19:24:17.902508 4507 raft_consensus.cc:374] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:17.902606 4507 raft_consensus.cc:729] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99328da9e1c49bf89b0a8a31225ed31, State: Initialized, Role: FOLLOWER
I20260810 19:24:17.903170 4507 consensus_queue.cc:260] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99328da9e1c49bf89b0a8a31225ed31" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42169 } }
I20260810 19:24:17.903411 4507 raft_consensus.cc:388] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:17.903548 4507 raft_consensus.cc:482] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:17.903702 4507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:17.905943 4507 raft_consensus.cc:504] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99328da9e1c49bf89b0a8a31225ed31" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42169 } }
I20260810 19:24:17.906443 4507 leader_election.cc:302] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b99328da9e1c49bf89b0a8a31225ed31; no voters:
I20260810 19:24:17.907061 4507 leader_election.cc:288] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:17.907184 4516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:17.908262 4516 raft_consensus.cc:686] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [term 1 LEADER]: Becoming Leader. State: Replica: b99328da9e1c49bf89b0a8a31225ed31, State: Running, Role: LEADER
I20260810 19:24:17.908869 4516 consensus_queue.cc:237] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99328da9e1c49bf89b0a8a31225ed31" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42169 } }
I20260810 19:24:17.909294 4507 sys_catalog.cc:564] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:17.911625 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b99328da9e1c49bf89b0a8a31225ed31. Latest consensus state: current_term: 1 leader_uuid: "b99328da9e1c49bf89b0a8a31225ed31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99328da9e1c49bf89b0a8a31225ed31" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42169 } } }
I20260810 19:24:17.912070 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:17.912654 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b99328da9e1c49bf89b0a8a31225ed31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99328da9e1c49bf89b0a8a31225ed31" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 42169 } } }
I20260810 19:24:17.912959 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:17.913662 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:17.918749 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:17.927688 4531 catalog_manager.cc:1347] Generated new cluster ID: 9fb4a30710ec4680940700e2a7cd9961
I20260810 19:24:17.927804 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:17.942095 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:17.943444 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:17.950403 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31: Generated new TSK 0
I20260810 19:24:17.951273 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:17.967231 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42169
--builtin_ntp_servers=127.7.43.20:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:24:18.318264 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:18.318581 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:18.318670 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:18.326026 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:18.326148 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:18.326257 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:24:18.335745 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:18.337502 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:18.339761 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:18.353372 4555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:18.353374 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:18.353572 4561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:18.353942 4547 server_base.cc:1034] running on GCE node
I20260810 19:24:18.354770 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:18.355789 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:18.357005 4547 hybrid_clock.cc:648] HybridClock initialized: now 1786389858356949 us; error 40 us; skew 500 ppm
I20260810 19:24:18.357314 4547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:18.359655 4547 webserver.cc:460] Webserver started at http://127.7.43.1:34999/ using document root <none> and password file <none>
I20260810 19:24:18.360363 4547 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:18.360453 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:18.360743 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:18.363849 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "32554a998b8941cfaf238a23dd68a8d0"
format_stamp: "Formatted at 2026-08-10 19:24:18 on dist-test-slave-7lbf"
server_key: "4f5132159b9ed8df5454af6c2a943633"
server_key_iv: "978571a5f648f77367c88ef52287e0af"
server_key_version: "encryptionkey@0"
I20260810 19:24:18.364686 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "32554a998b8941cfaf238a23dd68a8d0"
format_stamp: "Formatted at 2026-08-10 19:24:18 on dist-test-slave-7lbf"
server_key: "4f5132159b9ed8df5454af6c2a943633"
server_key_iv: "978571a5f648f77367c88ef52287e0af"
server_key_version: "encryptionkey@0"
I20260810 19:24:18.370231 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:18.374076 4568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:18.375502 4547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:24:18.375658 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "32554a998b8941cfaf238a23dd68a8d0"
format_stamp: "Formatted at 2026-08-10 19:24:18 on dist-test-slave-7lbf"
server_key: "4f5132159b9ed8df5454af6c2a943633"
server_key_iv: "978571a5f648f77367c88ef52287e0af"
server_key_version: "encryptionkey@0"
I20260810 19:24:18.375880 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:18.392226 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:18.393107 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:18.393395 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:18.394459 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:18.395987 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:18.396085 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:18.396191 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:18.396268 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:18.427620 4547 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:44229
I20260810 19:24:18.427670 4757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:44229 every 8 connection(s)
I20260810 19:24:18.428951 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:24:18.429958 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 4547
I20260810 19:24:18.430145 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:24:18.430670 7340 external_mini_cluster.cc:1428] Setting key 657b183fb1b4f2f57e7e854600be1c19
I20260810 19:24:18.433697 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42169
--builtin_ntp_servers=127.7.43.20:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:24:18.442533 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:18.431803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:47795 (local address 127.7.43.62:42169)
0810 19:24:18.432300 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:18.432307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:18.433822 (+ 1515us) server_negotiation.cc:407] Connection header received
0810 19:24:18.435724 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:18.435730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:18.435811 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:18.435935 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:18.437075 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.438053 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.439730 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.440272 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.440385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:18.441733 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:18.441739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:18.441743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:18.441765 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:18.441812 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:18.441827 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:18.441910 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:18.442115 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 19:24:18.442213 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260810 19:24:18.443538 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:18.432544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42169 (local address 127.7.43.1:47795)
0810 19:24:18.433673 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:18.433720 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:18.435447 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:18.436170 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:18.436178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:18.436270 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:18.436891 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:18.436912 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.438218 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.438222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:18.439581 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:18.439595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.440477 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.440481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:18.440697 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:18.441376 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:18.441379 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:18.441420 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:18.441503 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:18.441931 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:18.441938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:18.441944 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:24:18.442087 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 19:24:18.442406 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":324,"threads_started":1}
I20260810 19:24:18.445068 4759 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42169
I20260810 19:24:18.445369 4759 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:18.446175 4759 heartbeater.cc:511] Master 127.7.43.62:42169 requested a full tablet report, sending...
I20260810 19:24:18.448606 4454 ts_manager.cc:194] Registered new tserver with Master: 32554a998b8941cfaf238a23dd68a8d0 (127.7.43.1:44229)
I20260810 19:24:18.450767 4454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.1:47795
W20260810 19:24:18.783850 4764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:18.784180 4764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:18.784281 4764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:18.793489 4764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:18.793653 4764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:18.793776 4764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:24:18.803704 4764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:18.805087 4764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:18.806984 4764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:18.820320 4783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:18.820391 4774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:18.820269 4776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:18.821291 4764 server_base.cc:1034] running on GCE node
I20260810 19:24:18.821993 4764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:18.822989 4764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:18.824203 4764 hybrid_clock.cc:648] HybridClock initialized: now 1786389858824151 us; error 43 us; skew 500 ppm
I20260810 19:24:18.824530 4764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:18.826819 4764 webserver.cc:460] Webserver started at http://127.7.43.2:39327/ using document root <none> and password file <none>
I20260810 19:24:18.827548 4764 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:18.827641 4764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:18.827967 4764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:18.831043 4764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "a342bd8bedf2493790eb2b72e26f8bdb"
format_stamp: "Formatted at 2026-08-10 19:24:18 on dist-test-slave-7lbf"
server_key: "e09c1795d362012f71f4ad812dd824be"
server_key_iv: "d7cbe0249f0225735bc324c9f03b237d"
server_key_version: "encryptionkey@0"
I20260810 19:24:18.831833 4764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "a342bd8bedf2493790eb2b72e26f8bdb"
format_stamp: "Formatted at 2026-08-10 19:24:18 on dist-test-slave-7lbf"
server_key: "e09c1795d362012f71f4ad812dd824be"
server_key_iv: "d7cbe0249f0225735bc324c9f03b237d"
server_key_version: "encryptionkey@0"
I20260810 19:24:18.837183 4764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:18.841125 4794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:18.842587 4764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:18.842751 4764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "a342bd8bedf2493790eb2b72e26f8bdb"
format_stamp: "Formatted at 2026-08-10 19:24:18 on dist-test-slave-7lbf"
server_key: "e09c1795d362012f71f4ad812dd824be"
server_key_iv: "d7cbe0249f0225735bc324c9f03b237d"
server_key_version: "encryptionkey@0"
I20260810 19:24:18.842971 4764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:18.855404 4764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:18.856302 4764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:18.856585 4764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:18.857596 4764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:18.859139 4764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:18.859231 4764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:18.859341 4764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:18.859417 4764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:18.893204 4764 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:46325
I20260810 19:24:18.893245 4999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:46325 every 8 connection(s)
I20260810 19:24:18.894632 4764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:24:18.897770 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 4764
I20260810 19:24:18.897940 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:24:18.898377 7340 external_mini_cluster.cc:1428] Setting key cab63dbff9482b055bde87ab07f20e94
I20260810 19:24:18.901319 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42169
--builtin_ntp_servers=127.7.43.20:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:24:18.908046 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:18.897181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:37289 (local address 127.7.43.62:42169)
0810 19:24:18.897343 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:18.897347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:18.899155 (+ 1808us) server_negotiation.cc:407] Connection header received
0810 19:24:18.901387 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:18.901393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:18.901481 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:18.901631 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:18.902758 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.903758 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.905326 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.905872 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.905994 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:18.907300 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:18.907306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:18.907311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:18.907331 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:18.907369 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:18.907384 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:18.907467 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:18.907646 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 19:24:18.907750 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:24:18.909049 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:18.897721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42169 (local address 127.7.43.2:37289)
0810 19:24:18.898911 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:18.899003 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:18.901099 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:18.901851 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:18.901863 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:18.901975 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:18.902587 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:18.902604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.903890 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.903893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:18.905182 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:18.905195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:18.906067 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:18.906072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:18.906281 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:18.906967 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:18.906970 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:18.907017 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:18.907100 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:18.907466 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:18.907472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:18.907478 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:24:18.907624 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 19:24:18.907929 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":354,"threads_started":1}
I20260810 19:24:18.910467 5000 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42169
I20260810 19:24:18.910845 5000 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:18.911761 5000 heartbeater.cc:511] Master 127.7.43.62:42169 requested a full tablet report, sending...
I20260810 19:24:18.913362 4454 ts_manager.cc:194] Registered new tserver with Master: a342bd8bedf2493790eb2b72e26f8bdb (127.7.43.2:46325)
I20260810 19:24:18.914273 4454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.2:37289
W20260810 19:24:19.246140 5006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:19.246508 5006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:19.246670 5006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:19.253947 5006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:19.254047 5006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:19.254179 5006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:24:19.262996 5006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:42169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:19.264386 5006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:19.266206 5006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:19.279376 5019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:19.279464 5018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:19.279506 5021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:19.280030 5006 server_base.cc:1034] running on GCE node
I20260810 19:24:19.280697 5006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:19.281688 5006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:19.282888 5006 hybrid_clock.cc:648] HybridClock initialized: now 1786389859282825 us; error 56 us; skew 500 ppm
I20260810 19:24:19.283208 5006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:19.285431 5006 webserver.cc:460] Webserver started at http://127.7.43.3:35273/ using document root <none> and password file <none>
I20260810 19:24:19.286160 5006 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:19.286257 5006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:19.286571 5006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:19.289414 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "cca81565e7864940a06581f376c3ece5"
format_stamp: "Formatted at 2026-08-10 19:24:19 on dist-test-slave-7lbf"
server_key: "810ea7e4bf64fb14a0bcf0256a0c7d57"
server_key_iv: "7c1fca1bedab084d5a00c604b6702e22"
server_key_version: "encryptionkey@0"
I20260810 19:24:19.290230 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "cca81565e7864940a06581f376c3ece5"
format_stamp: "Formatted at 2026-08-10 19:24:19 on dist-test-slave-7lbf"
server_key: "810ea7e4bf64fb14a0bcf0256a0c7d57"
server_key_iv: "7c1fca1bedab084d5a00c604b6702e22"
server_key_version: "encryptionkey@0"
I20260810 19:24:19.295341 5006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:24:19.298900 5027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:19.300243 5006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:19.300398 5006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "cca81565e7864940a06581f376c3ece5"
format_stamp: "Formatted at 2026-08-10 19:24:19 on dist-test-slave-7lbf"
server_key: "810ea7e4bf64fb14a0bcf0256a0c7d57"
server_key_iv: "7c1fca1bedab084d5a00c604b6702e22"
server_key_version: "encryptionkey@0"
I20260810 19:24:19.300601 5006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:19.314599 5006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:19.315510 5006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:19.315793 5006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:19.316746 5006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:19.318411 5006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:19.318509 5006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:19.318617 5006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:19.318687 5006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:19.350140 5006 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:43693
I20260810 19:24:19.350229 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:43693 every 8 connection(s)
I20260810 19:24:19.351496 5006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:24:19.352948 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 5006
I20260810 19:24:19.353127 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:24:19.353703 7340 external_mini_cluster.cc:1428] Setting key ab248dce954ed13e8a96da0f4026577d
I20260810 19:24:19.364210 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.354101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:52379 (local address 127.7.43.62:42169)
0810 19:24:19.355243 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.355249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.355962 (+ 713us) server_negotiation.cc:407] Connection header received
0810 19:24:19.357928 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.357933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.358016 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.358101 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:19.359194 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.359934 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.361550 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.361969 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.362080 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.363413 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:19.363420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:19.363423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:19.363441 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:19.363474 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:19.363485 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:19.363573 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.363791 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.363893 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000}
I20260810 19:24:19.365227 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.354739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:42169 (local address 127.7.43.3:52379)
0810 19:24:19.355817 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.355862 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:19.357678 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:19.358358 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:19.358366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:19.358458 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:19.359036 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.359054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.360084 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.360088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.361374 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.361388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.362187 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.362194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.362423 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.363097 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:19.363099 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:19.363142 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:19.363223 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:19.363589 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:19.363601 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:19.363608 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 19:24:19.363766 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 19:24:19.364183 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":404,"threads_started":1}
I20260810 19:24:19.366634 5202 heartbeater.cc:347] Connected to a master server at 127.7.43.62:42169
I20260810 19:24:19.366933 5202 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:19.367689 5202 heartbeater.cc:511] Master 127.7.43.62:42169 requested a full tablet report, sending...
I20260810 19:24:19.369133 4454 ts_manager.cc:194] Registered new tserver with Master: cca81565e7864940a06581f376c3ece5 (127.7.43.3:43693)
I20260810 19:24:19.369964 4454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.3:52379
I20260810 19:24:19.381584 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:24:19.391709 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.385892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.7.43.62:42169)
0810 19:24:19.386152 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.386160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.386509 (+ 349us) server_negotiation.cc:407] Connection header received
0810 19:24:19.386681 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.386685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.386764 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.386842 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:19.387366 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.388128 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.389485 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.389940 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.390042 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.391120 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:19.391125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:19.391128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:19.391146 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:19.391179 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:19.391191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:19.391258 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.391342 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.391421 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:24:19.395723 4454 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:24:19.402740 4454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:24:19.425799 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.415461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:46325 (local address 127.0.0.1:52006)
0810 19:24:19.416775 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.416834 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:19.417140 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:19.418287 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:19.418294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:19.418326 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:19.418697 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.418718 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.420493 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.420497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.422450 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.422461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.424796 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.424803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.424987 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.425041 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 19:24:19.425171 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 19:24:19.425389 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":2065,"threads_started":1}
I20260810 19:24:19.425841 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.415539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.7.43.2:46325)
0810 19:24:19.416174 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.416184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.416962 (+ 778us) server_negotiation.cc:407] Connection header received
0810 19:24:19.417395 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.417403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.417723 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.417918 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:19.418893 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.420309 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.422636 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.424249 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.424381 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.424676 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.425252 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.425447 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":1032,"threads_started":1}
I20260810 19:24:19.426373 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.415963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:43693 (local address 127.0.0.1:52140)
0810 19:24:19.417817 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.417842 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:19.417948 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:19.419297 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:19.419302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:19.419330 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:19.419706 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.419718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.422072 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.422077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.424075 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.424089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.425490 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.425500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.425748 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.425780 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:24:19.425854 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 19:24:19.426033 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"mutex_wait_us":27,"thread_start_us":379,"threads_started":1}
I20260810 19:24:19.426790 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.416328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.7.43.3:43693)
0810 19:24:19.418693 (+ 2365us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.418701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.418736 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:24:19.418825 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.418832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.419082 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.419969 (+ 887us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:19.420276 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.421943 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.424294 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.425257 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.425980 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.426229 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.426284 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.426434 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2190,"thread_start_us":1958,"threads_started":1}
I20260810 19:24:19.430431 4889 tablet_service.cc:1467] Processing CreateTablet for tablet aa7eab6991804f4f90282f51b838bab1 (DEFAULT_TABLE table=test-table [id=5e2c7638423a42b7813e3dd6d6168797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:19.430433 5111 tablet_service.cc:1467] Processing CreateTablet for tablet aa7eab6991804f4f90282f51b838bab1 (DEFAULT_TABLE table=test-table [id=5e2c7638423a42b7813e3dd6d6168797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:19.431968 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7eab6991804f4f90282f51b838bab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:19.431965 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7eab6991804f4f90282f51b838bab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:19.435137 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.418396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.7.43.1:44229)
0810 19:24:19.418897 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.418905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.422091 (+ 3186us) server_negotiation.cc:407] Connection header received
0810 19:24:19.422415 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.422424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.422680 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.422883 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:19.423543 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.424990 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.431389 (+ 6399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.432870 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.432990 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.433458 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.434616 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.434793 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
I20260810 19:24:19.435582 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.417915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34476)
0810 19:24:19.421962 (+ 4047us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.421980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:19.422201 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:19.422888 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:19.422893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:19.422928 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:19.423355 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.423368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.425173 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.425178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.431147 (+ 5969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.431165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.433691 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.433698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.433880 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.433913 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:19.434003 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:24:19.434151 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3919,"mutex_wait_us":1,"thread_start_us":3785,"threads_started":1}
I20260810 19:24:19.439559 4656 tablet_service.cc:1467] Processing CreateTablet for tablet aa7eab6991804f4f90282f51b838bab1 (DEFAULT_TABLE table=test-table [id=5e2c7638423a42b7813e3dd6d6168797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:19.441113 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7eab6991804f4f90282f51b838bab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:19.444828 5234 tablet_bootstrap.cc:492] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Bootstrap starting.
I20260810 19:24:19.446873 5235 tablet_bootstrap.cc:492] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Bootstrap starting.
I20260810 19:24:19.448679 5234 tablet_bootstrap.cc:654] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:19.450057 5234 log.cc:824] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:19.450493 5235 tablet_bootstrap.cc:654] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:19.451735 5235 log.cc:824] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:19.453409 5234 tablet_bootstrap.cc:492] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: No bootstrap required, opened a new log
I20260810 19:24:19.453758 5234 ts_tablet_manager.cc:1397] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 19:24:19.454545 4759 heartbeater.cc:503] Master 127.7.43.62:42169 was elected leader, sending a full tablet report...
I20260810 19:24:19.454870 5235 tablet_bootstrap.cc:492] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: No bootstrap required, opened a new log
I20260810 19:24:19.455135 5235 ts_tablet_manager.cc:1397] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:24:19.462301 5241 tablet_bootstrap.cc:492] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: Bootstrap starting.
I20260810 19:24:19.464339 5234 raft_consensus.cc:348] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.465009 5234 raft_consensus.cc:374] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:19.465176 5234 raft_consensus.cc:729] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca81565e7864940a06581f376c3ece5, State: Initialized, Role: FOLLOWER
I20260810 19:24:19.465907 5241 tablet_bootstrap.cc:654] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:19.465708 5235 raft_consensus.cc:348] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.465950 5234 consensus_queue.cc:260] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.466180 5235 raft_consensus.cc:374] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:19.466315 5235 raft_consensus.cc:729] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a342bd8bedf2493790eb2b72e26f8bdb, State: Initialized, Role: FOLLOWER
I20260810 19:24:19.467106 5241 log.cc:824] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:19.466974 5235 consensus_queue.cc:260] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.468233 5000 heartbeater.cc:503] Master 127.7.43.62:42169 was elected leader, sending a full tablet report...
I20260810 19:24:19.468904 5235 ts_tablet_manager.cc:1428] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260810 19:24:19.469862 5202 heartbeater.cc:503] Master 127.7.43.62:42169 was elected leader, sending a full tablet report...
I20260810 19:24:19.472162 5234 ts_tablet_manager.cc:1428] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260810 19:24:19.472687 5241 tablet_bootstrap.cc:492] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: No bootstrap required, opened a new log
I20260810 19:24:19.472981 5241 ts_tablet_manager.cc:1397] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 19:24:19.480389 5241 raft_consensus.cc:348] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.480775 5241 raft_consensus.cc:374] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:19.480867 5241 raft_consensus.cc:729] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32554a998b8941cfaf238a23dd68a8d0, State: Initialized, Role: FOLLOWER
I20260810 19:24:19.481444 5241 consensus_queue.cc:260] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.483072 5241 ts_tablet_manager.cc:1428] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: Time spent starting tablet: real 0.010s user 0.007s sys 0.005s
I20260810 19:24:19.557891 5246 raft_consensus.cc:482] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:19.558224 5246 raft_consensus.cc:504] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.560011 5246 leader_election.cc:288] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a342bd8bedf2493790eb2b72e26f8bdb (127.7.43.2:46325), cca81565e7864940a06581f376c3ece5 (127.7.43.3:43693)
I20260810 19:24:19.566892 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.560472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:38259 (local address 127.7.43.3:43693)
0810 19:24:19.560688 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.560694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.561645 (+ 951us) server_negotiation.cc:407] Connection header received
0810 19:24:19.561783 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.561787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.561857 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.561950 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:19.562537 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.563228 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.565236 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.566044 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.566163 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.566258 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.566546 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.566661 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:24:19.566898 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.560440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:39657 (local address 127.7.43.2:46325)
0810 19:24:19.560645 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.560651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.561469 (+ 818us) server_negotiation.cc:407] Connection header received
0810 19:24:19.561663 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.561668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.561762 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.561840 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:19.562473 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.563179 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.565214 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.566024 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.566129 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.566223 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.566521 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.566647 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:24:19.566987 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.560440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:46325 (local address 127.7.43.1:39657)
0810 19:24:19.561348 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.561367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:19.561504 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:19.561870 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:19.561875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:19.561900 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:19.562322 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.562332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.563305 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.563311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.565010 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.565023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.566205 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.566211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.566357 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.566395 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 19:24:19.566466 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 19:24:19.566602 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":103,"thread_start_us":519,"threads_started":1}
I20260810 19:24:19.567070 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.560485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:43693 (local address 127.7.43.1:38259)
0810 19:24:19.561500 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.561538 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:19.561672 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:19.561985 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:19.561988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:19.562020 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:19.562359 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.562366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.563368 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.563372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.565011 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:19.565024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.566239 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.566246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:19.566368 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.566416 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 19:24:19.566501 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 19:24:19.566629 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":99,"spinlock_wait_cycles":768,"thread_start_us":586,"threads_started":1}
I20260810 19:24:19.568118 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7eab6991804f4f90282f51b838bab1" candidate_uuid: "32554a998b8941cfaf238a23dd68a8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" is_pre_election: true
I20260810 19:24:19.568118 5139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7eab6991804f4f90282f51b838bab1" candidate_uuid: "32554a998b8941cfaf238a23dd68a8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca81565e7864940a06581f376c3ece5" is_pre_election: true
I20260810 19:24:19.568733 5139 raft_consensus.cc:2454] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32554a998b8941cfaf238a23dd68a8d0 in term 0.
I20260810 19:24:19.568727 4924 raft_consensus.cc:2454] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32554a998b8941cfaf238a23dd68a8d0 in term 0.
I20260810 19:24:19.569442 4569 leader_election.cc:302] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32554a998b8941cfaf238a23dd68a8d0, cca81565e7864940a06581f376c3ece5; no voters:
I20260810 19:24:19.569886 5246 raft_consensus.cc:2781] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:24:19.570016 5246 raft_consensus.cc:482] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:24:19.570127 5246 raft_consensus.cc:3037] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:19.572372 5246 raft_consensus.cc:504] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.573029 5246 leader_election.cc:288] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [CANDIDATE]: Term 1 election: Requested vote from peers a342bd8bedf2493790eb2b72e26f8bdb (127.7.43.2:46325), cca81565e7864940a06581f376c3ece5 (127.7.43.3:43693)
I20260810 19:24:19.573498 4924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7eab6991804f4f90282f51b838bab1" candidate_uuid: "32554a998b8941cfaf238a23dd68a8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a342bd8bedf2493790eb2b72e26f8bdb"
I20260810 19:24:19.573575 5139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7eab6991804f4f90282f51b838bab1" candidate_uuid: "32554a998b8941cfaf238a23dd68a8d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca81565e7864940a06581f376c3ece5"
I20260810 19:24:19.573734 4924 raft_consensus.cc:3037] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:19.573792 5139 raft_consensus.cc:3037] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:19.575965 4924 raft_consensus.cc:2454] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32554a998b8941cfaf238a23dd68a8d0 in term 1.
I20260810 19:24:19.575973 5139 raft_consensus.cc:2454] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32554a998b8941cfaf238a23dd68a8d0 in term 1.
I20260810 19:24:19.576390 4572 leader_election.cc:302] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32554a998b8941cfaf238a23dd68a8d0, a342bd8bedf2493790eb2b72e26f8bdb, cca81565e7864940a06581f376c3ece5; no voters:
I20260810 19:24:19.576690 5246 raft_consensus.cc:2781] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:19.577109 5246 raft_consensus.cc:686] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 1 LEADER]: Becoming Leader. State: Replica: 32554a998b8941cfaf238a23dd68a8d0, State: Running, Role: LEADER
I20260810 19:24:19.577649 5246 consensus_queue.cc:237] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } }
I20260810 19:24:19.582415 4453 catalog_manager.cc:5495] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32554a998b8941cfaf238a23dd68a8d0 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "32554a998b8941cfaf238a23dd68a8d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32554a998b8941cfaf238a23dd68a8d0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 44229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 } health_report { overall_health: UNKNOWN } } }
W20260810 19:24:19.603220 5204 tablet.cc:2369] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:24:19.623008 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.617913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.7.43.1:44229)
0810 19:24:19.618140 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.618147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.618185 (+ 38us) server_negotiation.cc:407] Connection header received
0810 19:24:19.618418 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.618422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.618519 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.618645 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:24:19.618990 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.619664 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.620984 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.621413 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.621508 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.622030 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:24:19.622523 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:24:19.622639 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.622675 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.622736 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:24:19.631366 4924 raft_consensus.cc:1261] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 1 FOLLOWER]: Refusing update from remote peer 32554a998b8941cfaf238a23dd68a8d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:19.631392 5139 raft_consensus.cc:1261] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 1 FOLLOWER]: Refusing update from remote peer 32554a998b8941cfaf238a23dd68a8d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:19.632319 5249 consensus_queue.cc:1035] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca81565e7864940a06581f376c3ece5" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 43693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:19.632688 5246 consensus_queue.cc:1035] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a342bd8bedf2493790eb2b72e26f8bdb" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:19.647837 5269 mvcc.cc:205] Tried to move back new op lower bound from 7317052865039458304 to 7317052864833654784. Current Snapshot: MvccSnapshot[applied={T|T < 7317052865039458304}]
I20260810 19:24:19.659857 4453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:19.660138 4453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:19.663802 4453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b99328da9e1c49bf89b0a8a31225ed31: Sending DeleteTablet for 3 replicas of tablet aa7eab6991804f4f90282f51b838bab1
I20260810 19:24:19.665127 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7eab6991804f4f90282f51b838bab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:19 UTC) from {username='slave'} at 127.0.0.1:34476
I20260810 19:24:19.665114 5111 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7eab6991804f4f90282f51b838bab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:19 UTC) from {username='slave'} at 127.0.0.1:52140
I20260810 19:24:19.665166 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7eab6991804f4f90282f51b838bab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:19 UTC) from {username='slave'} at 127.0.0.1:52006
I20260810 19:24:19.667219 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 4547
I20260810 19:24:19.677596 5281 tablet_replica.cc:330] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: stopping tablet replica
I20260810 19:24:19.678335 5282 tablet_replica.cc:330] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: stopping tablet replica
I20260810 19:24:19.679004 5282 raft_consensus.cc:2229] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:24:19.679453 5284 tablet_replica.cc:330] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: stopping tablet replica
I20260810 19:24:19.679672 5281 raft_consensus.cc:2229] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:19.679883 5284 raft_consensus.cc:2229] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:19.680531 5284 raft_consensus.cc:2258] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:19.680924 5281 raft_consensus.cc:2258] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:19.681402 5282 raft_consensus.cc:2258] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:19.683563 5281 ts_tablet_manager.cc:1905] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:19.684113 5282 ts_tablet_manager.cc:1905] T aa7eab6991804f4f90282f51b838bab1 P 32554a998b8941cfaf238a23dd68a8d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:19.687124 5284 ts_tablet_manager.cc:1905] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:19.689023 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.670023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.7.43.1:44229)
0810 19:24:19.670527 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.670538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.678721 (+ 8183us) server_negotiation.cc:407] Connection header received
0810 19:24:19.678997 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.679003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.679118 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.679240 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:19.679986 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.682474 (+ 2488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.684496 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.685178 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.685293 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.686910 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:19.686922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:19.686940 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:19.687728 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:19.687799 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:19.687825 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:19.687933 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.688503 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.688616 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371}
I20260810 19:24:19.690078 4753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:19.691404 5281 ts_tablet_manager.cc:1918] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:19.691622 5281 log.cc:1196] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/aa7eab6991804f4f90282f51b838bab1
I20260810 19:24:19.692199 5281 ts_tablet_manager.cc:1939] T aa7eab6991804f4f90282f51b838bab1 P cca81565e7864940a06581f376c3ece5: Deleting consensus metadata
I20260810 19:24:19.694046 5284 ts_tablet_manager.cc:1918] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:19.694276 5284 log.cc:1196] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/aa7eab6991804f4f90282f51b838bab1
I20260810 19:24:19.694311 4417 catalog_manager.cc:4841] TS cca81565e7864940a06581f376c3ece5 (127.7.43.3:43693): tablet aa7eab6991804f4f90282f51b838bab1 (table test-table [id=5e2c7638423a42b7813e3dd6d6168797]) successfully deleted
I20260810 19:24:19.694761 5284 ts_tablet_manager.cc:1939] T aa7eab6991804f4f90282f51b838bab1 P a342bd8bedf2493790eb2b72e26f8bdb: Deleting consensus metadata
I20260810 19:24:19.696239 4422 catalog_manager.cc:4841] TS a342bd8bedf2493790eb2b72e26f8bdb (127.7.43.2:46325): tablet aa7eab6991804f4f90282f51b838bab1 (table test-table [id=5e2c7638423a42b7813e3dd6d6168797]) successfully deleted
I20260810 19:24:19.787302 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 4547
W20260810 19:24:19.815001 4422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:44229 (error 108)
I20260810 19:24:19.815256 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 4764
I20260810 19:24:19.816982 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.816492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34508)
0810 19:24:19.816672 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.816807 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 19:24:19.817229 4422 catalog_manager.cc:4584] TS 32554a998b8941cfaf238a23dd68a8d0 (127.7.43.1:44229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa7eab6991804f4f90282f51b838bab1: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
I20260810 19:24:19.822312 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.816059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.7.43.2:46325)
0810 19:24:19.816228 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.816233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.816255 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:24:19.816486 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.816493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.816605 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.816720 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:19.817217 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.818136 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.819479 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.819892 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.819985 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.820938 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:19.820947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:19.820962 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:19.821679 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:19.821731 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:19.821753 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:19.821833 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.821957 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.822041 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:24:19.823000 4992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:19.871156 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.870562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34516)
0810 19:24:19.870822 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.870951 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 19:24:19.872355 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.871939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34532)
0810 19:24:19.872098 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.872200 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:24:19.911574 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 4764
I20260810 19:24:19.932968 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.932504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34538)
0810 19:24:19.932719 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.932810 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 19:24:19.934461 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.934036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34554)
0810 19:24:19.934229 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:19.934332 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:24:19.938297 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 5006
I20260810 19:24:19.945308 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:19.939147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.7.43.3:43693)
0810 19:24:19.939299 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:19.939304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:19.939322 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:24:19.939407 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:19.939411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:19.939518 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:19.939599 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:19.940082 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.940880 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.942352 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:19.942751 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:19.942842 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:19.944049 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:19.944059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:19.944075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:19.944656 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:19.944727 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:19.944748 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:19.944843 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:19.944969 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 19:24:19.945060 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:24:19.946038 5194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:20.015447 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:20.014967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34564)
0810 19:24:20.015170 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:20.015260 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 19:24:20.016628 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:20.016211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:44229 (local address 127.0.0.1:34572)
0810 19:24:20.016433 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:20.016509 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:44229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 19:24:20.033571 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 5006
I20260810 19:24:20.058234 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 4389
I20260810 19:24:20.059386 4501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:20.168803 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 4389
2026-08-10T19:24:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2871 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:20 dist-test-slave-7lbf krb5kdc[5298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:20 dist-test-slave-7lbf krb5kdc[5298](info): set up 2 sockets
Aug 10 19:24:20 dist-test-slave-7lbf krb5kdc[5298](info): commencing operation
krb5kdc: starting...
W20260810 19:24:21.487092 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:21 dist-test-slave-7lbf krb5kdc[5298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:24:21.804658 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35595
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:22.149659 5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:22.149951 5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:22.150085 5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:22.157207 5319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:22.157316 5319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:22.157398 5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:22.157475 5319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:22.157557 5319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:22.167377 5319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:22.168697 5319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:22.170102 5319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:22.181770 5325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:22.181738 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:22.181690 5331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:22.182871 5319 server_base.cc:1034] running on GCE node
I20260810 19:24:22.183615 5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:22.184962 5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:22.186159 5319 hybrid_clock.cc:648] HybridClock initialized: now 1786389862186100 us; error 50 us; skew 500 ppm
I20260810 19:24:22.186499 5319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:22 dist-test-slave-7lbf krb5kdc[5298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389862, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:22.192902 5319 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:24:22.194403 5319 webserver.cc:460] Webserver started at http://127.7.43.62:42803/ using document root <none> and password file <none>
I20260810 19:24:22.195106 5319 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:22.195223 5319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:22.195525 5319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:22.198582 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "abbba351064e4a0d9b9dd4b8577fc120"
format_stamp: "Formatted at 2026-08-10 19:24:22 on dist-test-slave-7lbf"
server_key: "32a45c269a28f4457f31d5d9b6ff63d8"
server_key_iv: "3b8a71e2e67c88bf63845e14e99e909d"
server_key_version: "encryptionkey@0"
I20260810 19:24:22.199333 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "abbba351064e4a0d9b9dd4b8577fc120"
format_stamp: "Formatted at 2026-08-10 19:24:22 on dist-test-slave-7lbf"
server_key: "32a45c269a28f4457f31d5d9b6ff63d8"
server_key_iv: "3b8a71e2e67c88bf63845e14e99e909d"
server_key_version: "encryptionkey@0"
I20260810 19:24:22.204560 5319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:24:22.208119 5339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:22.209399 5319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:24:22.209632 5319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "abbba351064e4a0d9b9dd4b8577fc120"
format_stamp: "Formatted at 2026-08-10 19:24:22 on dist-test-slave-7lbf"
server_key: "32a45c269a28f4457f31d5d9b6ff63d8"
server_key_iv: "3b8a71e2e67c88bf63845e14e99e909d"
server_key_version: "encryptionkey@0"
I20260810 19:24:22.209822 5319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:22.224004 5319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:22.228286 5319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:22.228567 5319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:22.247089 5319 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:35595
I20260810 19:24:22.247134 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:35595 every 8 connection(s)
I20260810 19:24:22.248464 5319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:22.252278 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:22.253786 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 5319
I20260810 19:24:22.253955 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:22.254410 7340 external_mini_cluster.cc:1428] Setting key 188e760cb002de6f551bfff39cd549f2
I20260810 19:24:22.266480 5426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120: Bootstrap starting.
Aug 10 19:24:22 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:22.269773 5426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:22.271046 5426 log.cc:824] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:22.273945 5426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120: No bootstrap required, opened a new log
I20260810 19:24:22.277812 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:22.256273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.7.43.62:35595)
0810 19:24:22.257139 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:22.257153 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:22.257213 (+ 60us) server_negotiation.cc:407] Connection header received
0810 19:24:22.258984 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:22.259014 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:22.259541 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:22.260012 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:22.261307 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:22.263351 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:22.264680 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:22.265724 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:22.265921 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:22.270362 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:22.270374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:22.270393 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:22.270438 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:22.273662 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:22.274420 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:22.274426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:22.274433 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:22.274608 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:22.274983 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:22.274987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:22.274989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:22.275536 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:22.275863 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:22.276321 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 19:24:22.276733 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":367,"threads_started":1}
I20260810 19:24:22.281885 5426 raft_consensus.cc:348] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbba351064e4a0d9b9dd4b8577fc120" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35595 } }
I20260810 19:24:22.282357 5426 raft_consensus.cc:374] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:22.282471 5426 raft_consensus.cc:729] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbba351064e4a0d9b9dd4b8577fc120, State: Initialized, Role: FOLLOWER
I20260810 19:24:22.283047 5426 consensus_queue.cc:260] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbba351064e4a0d9b9dd4b8577fc120" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35595 } }
I20260810 19:24:22.283290 5426 raft_consensus.cc:388] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:22.283427 5426 raft_consensus.cc:482] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:22.283579 5426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:22.285733 5426 raft_consensus.cc:504] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbba351064e4a0d9b9dd4b8577fc120" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35595 } }
I20260810 19:24:22.286219 5426 leader_election.cc:302] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abbba351064e4a0d9b9dd4b8577fc120; no voters:
I20260810 19:24:22.286763 5426 leader_election.cc:288] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:22.286886 5434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:22.288470 5434 raft_consensus.cc:686] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [term 1 LEADER]: Becoming Leader. State: Replica: abbba351064e4a0d9b9dd4b8577fc120, State: Running, Role: LEADER
I20260810 19:24:22.288967 5434 consensus_queue.cc:237] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbba351064e4a0d9b9dd4b8577fc120" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35595 } }
I20260810 19:24:22.289196 5426 sys_catalog.cc:564] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:22.291944 5436 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abbba351064e4a0d9b9dd4b8577fc120. Latest consensus state: current_term: 1 leader_uuid: "abbba351064e4a0d9b9dd4b8577fc120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbba351064e4a0d9b9dd4b8577fc120" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35595 } } }
I20260810 19:24:22.292097 5435 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abbba351064e4a0d9b9dd4b8577fc120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbba351064e4a0d9b9dd4b8577fc120" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35595 } } }
I20260810 19:24:22.292405 5436 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:22.292405 5435 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:22.294052 5459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:22.299134 5459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:22.308911 5459 catalog_manager.cc:1347] Generated new cluster ID: ba174558445c43778a9064fd14628c09
I20260810 19:24:22.309051 5459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:22.324070 5459 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:22.325443 5459 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:22.335188 5459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120: Generated new TSK 0
I20260810 19:24:22.336043 5459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:24:22.413312 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:35595
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:22.758759 5474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:22.759138 5474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:22.759262 5474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:22.766539 5474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:22.766650 5474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:22.766803 5474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:24:22.776563 5474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:22.778004 5474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:22.779456 5474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:22.795467 5486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:22.795392 5489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:22.795432 5487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:22.796371 5474 server_base.cc:1034] running on GCE node
I20260810 19:24:22.797040 5474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:22.798121 5474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:22.799350 5474 hybrid_clock.cc:648] HybridClock initialized: now 1786389862799266 us; error 63 us; skew 500 ppm
I20260810 19:24:22.799690 5474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:22 dist-test-slave-7lbf krb5kdc[5298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389862, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:22.803023 5474 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:24:22.804557 5474 webserver.cc:460] Webserver started at http://127.7.43.1:42773/ using document root <none> and password file <none>
I20260810 19:24:22.805275 5474 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:22.805398 5474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:22.805768 5474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:22.808743 5474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "4bfddd73705045ee867d5d8870441dc3"
format_stamp: "Formatted at 2026-08-10 19:24:22 on dist-test-slave-7lbf"
server_key: "57a94a6d179d24908836222f2c654ca4"
server_key_iv: "ff9802a37bbbd23f4aaef7a01a163463"
server_key_version: "encryptionkey@0"
I20260810 19:24:22.809576 5474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "4bfddd73705045ee867d5d8870441dc3"
format_stamp: "Formatted at 2026-08-10 19:24:22 on dist-test-slave-7lbf"
server_key: "57a94a6d179d24908836222f2c654ca4"
server_key_iv: "ff9802a37bbbd23f4aaef7a01a163463"
server_key_version: "encryptionkey@0"
I20260810 19:24:22.815086 5474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 19:24:22.818778 5501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:22.820219 5474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:22.820375 5474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "4bfddd73705045ee867d5d8870441dc3"
format_stamp: "Formatted at 2026-08-10 19:24:22 on dist-test-slave-7lbf"
server_key: "57a94a6d179d24908836222f2c654ca4"
server_key_iv: "ff9802a37bbbd23f4aaef7a01a163463"
server_key_version: "encryptionkey@0"
I20260810 19:24:22.820585 5474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:22.832279 5474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:22.836311 5474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:22.836589 5474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:22.837584 5474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:22.838927 5474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:22.839007 5474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:22.839128 5474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:22.839181 5474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:22.868417 5474 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:38909
I20260810 19:24:22.868467 5691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:38909 every 8 connection(s)
I20260810 19:24:22.869761 5474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:24:22.878324 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 5474
I20260810 19:24:22.878530 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:24:22.879030 7340 external_mini_cluster.cc:1428] Setting key 7d8360473db70ebaa21c0805064f668e
Aug 10 19:24:22 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389862, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:22.890712 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:22.872430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:40645 (local address 127.7.43.62:35595)
0810 19:24:22.872852 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:22.872856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:22.873872 (+ 1016us) server_negotiation.cc:407] Connection header received
0810 19:24:22.875763 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:22.875769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:22.875850 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:22.875948 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:22.877497 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:22.878414 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:22.880280 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:22.880768 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:22.880885 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:22.884515 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:22.884523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:22.884527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:22.884567 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:22.887519 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:22.888257 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:22.888262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:22.888265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:22.888420 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:22.888808 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:22.888812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:22.888815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:22.889165 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:22.889310 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:22.890096 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 19:24:22.890319 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20260810 19:24:22.891666 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:22.872952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:35595 (local address 127.7.43.1:40645)
0810 19:24:22.873712 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:22.873753 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:22.875489 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:22.876196 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:22.876205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:22.876736 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:22.877322 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:22.877340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:22.878598 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:22.878604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:22.880133 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:22.880152 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:22.881010 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:22.881018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:22.881297 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:22.882017 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:22.882042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:22.884256 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:22.887672 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:22.887679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:22.887695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:22.888134 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:22.888516 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:22.888520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:22.888522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:22.888710 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:22.889281 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:22.889288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:22.889692 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 19:24:22.889963 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 19:24:22.890434 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":290,"threads_started":1}
I20260810 19:24:22.893280 5692 heartbeater.cc:347] Connected to a master server at 127.7.43.62:35595
I20260810 19:24:22.893678 5692 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:22.894532 5692 heartbeater.cc:511] Master 127.7.43.62:35595 requested a full tablet report, sending...
I20260810 19:24:22.896734 5365 ts_manager.cc:194] Registered new tserver with Master: 4bfddd73705045ee867d5d8870441dc3 (127.7.43.1:38909)
I20260810 19:24:22.898293 5365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:40645
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:24:22.948781 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:35595
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:23.284322 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:23.284632 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:23.284771 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:23.292229 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:23.292356 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:23.292510 5701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:24:23.303109 5701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:23.304507 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:23.305949 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:23.319228 5709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:23.319276 5712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:23.319240 5710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:23.320585 5701 server_base.cc:1034] running on GCE node
I20260810 19:24:23.321199 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:23.322180 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:23.323379 5701 hybrid_clock.cc:648] HybridClock initialized: now 1786389863323323 us; error 50 us; skew 500 ppm
I20260810 19:24:23.323694 5701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:23 dist-test-slave-7lbf krb5kdc[5298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389863, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:23.327008 5701 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:24:23.328644 5701 webserver.cc:460] Webserver started at http://127.7.43.2:35431/ using document root <none> and password file <none>
I20260810 19:24:23.329339 5701 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:23.329460 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:23.329803 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:23.332794 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "74e943e7796f43e98fec1d0e367756b0"
format_stamp: "Formatted at 2026-08-10 19:24:23 on dist-test-slave-7lbf"
server_key: "d535f58c3dc3d81307cd4114ce076524"
server_key_iv: "6da9892cff9f1133c1058e9d9d251633"
server_key_version: "encryptionkey@0"
I20260810 19:24:23.333619 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "74e943e7796f43e98fec1d0e367756b0"
format_stamp: "Formatted at 2026-08-10 19:24:23 on dist-test-slave-7lbf"
server_key: "d535f58c3dc3d81307cd4114ce076524"
server_key_iv: "6da9892cff9f1133c1058e9d9d251633"
server_key_version: "encryptionkey@0"
I20260810 19:24:23.338801 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:24:23.342482 5720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:23.343820 5701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:23.343983 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "74e943e7796f43e98fec1d0e367756b0"
format_stamp: "Formatted at 2026-08-10 19:24:23 on dist-test-slave-7lbf"
server_key: "d535f58c3dc3d81307cd4114ce076524"
server_key_iv: "6da9892cff9f1133c1058e9d9d251633"
server_key_version: "encryptionkey@0"
I20260810 19:24:23.344208 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:23.356925 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:23.360978 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:23.361261 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:23.362274 5701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:23.363664 5701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:23.363742 5701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:23.363862 5701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:23.363916 5701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:23.395581 5701 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:38787
I20260810 19:24:23.395653 5913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:38787 every 8 connection(s)
I20260810 19:24:23.396885 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:24:23.399497 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 5701
I20260810 19:24:23.399675 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:24:23.400215 7340 external_mini_cluster.cc:1428] Setting key ff1fdfa617e9f2392de76b3ee42d4f0e
Aug 10 19:24:23 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389863, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:23.417949 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:23.399491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:44185 (local address 127.7.43.62:35595)
0810 19:24:23.400050 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:23.400057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:23.401107 (+ 1050us) server_negotiation.cc:407] Connection header received
0810 19:24:23.403444 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:23.403450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:23.403533 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:23.403633 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:23.405257 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.406055 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.407625 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.408073 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.408186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:23.412060 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:23.412065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:23.412069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:23.412104 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:23.414749 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:23.415572 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:23.415576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:23.415578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:23.415703 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:23.416180 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:23.416185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:23.416188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:23.416533 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:23.416681 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:23.417331 (+ 650us) server_negotiation.cc:299] Negotiation successful
0810 19:24:23.417584 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
I20260810 19:24:23.418910 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:23.400091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:35595 (local address 127.7.43.2:44185)
0810 19:24:23.400932 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:23.400981 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:23.403168 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:23.403904 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:23.403914 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:23.404469 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:23.405064 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:23.405082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.406201 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.406205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:23.407483 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:23.407496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.408301 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.408309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:23.408535 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:23.409288 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:23.409313 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:23.411809 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:23.414899 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:23.414908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:23.414924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:23.415475 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:23.415798 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:23.415802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:23.415806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:23.416055 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:23.416642 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:23.416648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:23.416968 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 19:24:23.417212 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 19:24:23.417657 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":314,"threads_started":1}
I20260810 19:24:23.420372 5914 heartbeater.cc:347] Connected to a master server at 127.7.43.62:35595
I20260810 19:24:23.420765 5914 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:23.421653 5914 heartbeater.cc:511] Master 127.7.43.62:35595 requested a full tablet report, sending...
I20260810 19:24:23.423249 5365 ts_manager.cc:194] Registered new tserver with Master: 74e943e7796f43e98fec1d0e367756b0 (127.7.43.2:38787)
I20260810 19:24:23.424134 5365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:44185
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:24:23.470875 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:35595
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:23.824801 5924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:23.825141 5924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:23.825269 5924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:23.833195 5924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:23.833323 5924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:23.833456 5924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:24:23.843624 5924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:23.844988 5924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:23.846372 5924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:23.860114 5930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:23.860060 5931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:23.860340 5933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:23.860899 5924 server_base.cc:1034] running on GCE node
I20260810 19:24:23.861583 5924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:23.862565 5924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:23.863768 5924 hybrid_clock.cc:648] HybridClock initialized: now 1786389863863716 us; error 47 us; skew 500 ppm
I20260810 19:24:23.864092 5924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:23 dist-test-slave-7lbf krb5kdc[5298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389863, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:23.867357 5924 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:24:23.869318 5924 webserver.cc:460] Webserver started at http://127.7.43.3:45071/ using document root <none> and password file <none>
I20260810 19:24:23.870256 5924 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:23.870404 5924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:23.870774 5924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:23.874832 5924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "b06be987decb4d0fbc090f2ede80c718"
format_stamp: "Formatted at 2026-08-10 19:24:23 on dist-test-slave-7lbf"
server_key: "bf43f4595df9597463c72a8f0fe9fc5e"
server_key_iv: "71ae43ebc2c116b7207d9a1fe9e31a4c"
server_key_version: "encryptionkey@0"
I20260810 19:24:23.875588 5924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "b06be987decb4d0fbc090f2ede80c718"
format_stamp: "Formatted at 2026-08-10 19:24:23 on dist-test-slave-7lbf"
server_key: "bf43f4595df9597463c72a8f0fe9fc5e"
server_key_iv: "71ae43ebc2c116b7207d9a1fe9e31a4c"
server_key_version: "encryptionkey@0"
I20260810 19:24:23.881031 5924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 19:24:23.884563 5947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:23.885944 5924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:23.886102 5924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "b06be987decb4d0fbc090f2ede80c718"
format_stamp: "Formatted at 2026-08-10 19:24:23 on dist-test-slave-7lbf"
server_key: "bf43f4595df9597463c72a8f0fe9fc5e"
server_key_iv: "71ae43ebc2c116b7207d9a1fe9e31a4c"
server_key_version: "encryptionkey@0"
I20260810 19:24:23.886283 5924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:23.901829 5692 heartbeater.cc:503] Master 127.7.43.62:35595 was elected leader, sending a full tablet report...
I20260810 19:24:23.909267 5924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:23.913810 5924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:23.914103 5924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:23.915138 5924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:23.916471 5924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:23.916553 5924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:23.916674 5924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:23.916729 5924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:23.948334 5924 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:37719
I20260810 19:24:23.948442 6120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:37719 every 8 connection(s)
I20260810 19:24:23.949725 5924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:24:23.953914 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 5924
I20260810 19:24:23.954118 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:24:23.954658 7340 external_mini_cluster.cc:1428] Setting key 9569de7377d3735e49ed00a525c3d674
Aug 10 19:24:23 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389863, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:23.969674 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:23.952445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:43987 (local address 127.7.43.62:35595)
0810 19:24:23.952971 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:23.952979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:23.953973 (+ 994us) server_negotiation.cc:407] Connection header received
0810 19:24:23.955913 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:23.955919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:23.956002 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:23.956095 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:23.957887 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.958676 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.960189 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.960596 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.960705 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:23.963997 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:23.964003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:23.964007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:23.964039 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:23.966574 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:23.967281 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:23.967285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:23.967288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:23.967410 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:23.967820 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:23.967824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:23.967826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:23.968108 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:23.968243 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:23.969019 (+ 776us) server_negotiation.cc:299] Negotiation successful
0810 19:24:23.969248 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":232,"threads_started":1}
I20260810 19:24:23.970650 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:23.953063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:35595 (local address 127.7.43.3:43987)
0810 19:24:23.953818 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:23.953863 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:23.955617 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:23.956341 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:23.956349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:23.956981 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:23.957732 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:23.957752 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.958822 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.958826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:23.960056 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:23.960069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.960808 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.960816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:23.961044 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:23.961675 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:23.961698 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:23.963800 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:23.966704 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:23.966713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:23.966729 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:23.967155 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:23.967511 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:23.967516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:23.967520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:23.967704 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:23.968238 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:23.968246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:23.968605 (+ 359us) client_negotiation.cc:769] Sending connection context
0810 19:24:23.968885 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 19:24:23.969427 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":302,"threads_started":1}
I20260810 19:24:23.972123 6121 heartbeater.cc:347] Connected to a master server at 127.7.43.62:35595
I20260810 19:24:23.972447 6121 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:23.973169 6121 heartbeater.cc:511] Master 127.7.43.62:35595 requested a full tablet report, sending...
I20260810 19:24:23.974649 5365 ts_manager.cc:194] Registered new tserver with Master: b06be987decb4d0fbc090f2ede80c718 (127.7.43.3:37719)
I20260810 19:24:23.975493 5365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:43987
I20260810 19:24:23.982550 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:24:24.000006 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:23.986692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.7.43.62:35595)
0810 19:24:23.986865 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:23.986869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:23.987327 (+ 458us) server_negotiation.cc:407] Connection header received
0810 19:24:23.987485 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:23.987504 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:23.987572 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:23.987653 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:23.989118 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.990086 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.991449 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:23.991911 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:23.992019 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:23.994664 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:23.994670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:23.994673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:23.994708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:23.997306 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:23.998119 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:23.998123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:23.998126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:23.998260 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:23.998621 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:23.998624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:23.998626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:23.998981 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:23.999200 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:23.999456 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 19:24:23.999607 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:24:24.003970 5365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:24:24.011251 5365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:24:24.035461 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.023857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:37719 (local address 127.0.0.1:38178)
0810 19:24:24.025667 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.025705 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:24.025873 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:24.028068 (+ 2195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:24.028076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:24.028104 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:24.028710 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.028725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.030781 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.030786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.033037 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.033048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.034392 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.034398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.034595 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.034636 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 19:24:24.034772 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 19:24:24.034984 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"mutex_wait_us":41,"thread_start_us":2252,"threads_started":1}
I20260810 19:24:24.036110 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.025936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.7.43.2:38787)
0810 19:24:24.026505 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.026515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.027160 (+ 645us) server_negotiation.cc:407] Connection header received
0810 19:24:24.027332 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.027349 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.027851 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.028050 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:24.029301 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.030377 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.033175 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.034600 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.034721 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.034882 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.035603 (+ 721us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.035784 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":262,"threads_started":1}
I20260810 19:24:24.036155 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.023880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52686)
0810 19:24:24.026396 (+ 2516us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.026412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:24.026532 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:24.028288 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:24.028293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:24.028316 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:24.028921 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.028933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.030823 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.030828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.032823 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.032836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.034964 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.034970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.035230 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.035263 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:24.035358 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:24:24.035510 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2388,"mutex_wait_us":132,"thread_start_us":1380,"threads_started":1}
I20260810 19:24:24.036178 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.025634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:38787 (local address 127.0.0.1:35486)
0810 19:24:24.027030 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.027046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:24.027169 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:24.028301 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:24.028304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:24.028327 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:24.029173 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.029180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.030510 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.030515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.032988 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.033008 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.034802 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.034808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.035021 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.035086 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 19:24:24.035179 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:24:24.035308 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":404,"thread_start_us":1114,"threads_started":1}
I20260810 19:24:24.037026 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.024463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.7.43.3:37719)
0810 19:24:24.027035 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.027045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.027082 (+ 37us) server_negotiation.cc:407] Connection header received
0810 19:24:24.027550 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.027558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.027812 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.028683 (+ 871us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:24.028954 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.030624 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.033195 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.034189 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.034308 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.034552 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.036491 (+ 1939us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.036672 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2388,"thread_start_us":2217,"threads_started":1}
I20260810 19:24:24.037205 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.023999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.7.43.1:38909)
0810 19:24:24.027736 (+ 3737us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.027743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.027779 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:24:24.027876 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.027883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.028082 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.028422 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:24.029087 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.030673 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.032976 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.034447 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.034579 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.034811 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.036745 (+ 1934us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.036909 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3572,"thread_start_us":3399,"threads_started":1}
I20260810 19:24:24.039587 5801 tablet_service.cc:1467] Processing CreateTablet for tablet ef88d62e17bb4daca9de0818926d0b2d (DEFAULT_TABLE table=test-table [id=b2a7a4a3caa24c2a8db330ca16b79915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:24.040122 5579 tablet_service.cc:1467] Processing CreateTablet for tablet ef88d62e17bb4daca9de0818926d0b2d (DEFAULT_TABLE table=test-table [id=b2a7a4a3caa24c2a8db330ca16b79915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:24.041035 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef88d62e17bb4daca9de0818926d0b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:24.041616 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef88d62e17bb4daca9de0818926d0b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:24.043689 6013 tablet_service.cc:1467] Processing CreateTablet for tablet ef88d62e17bb4daca9de0818926d0b2d (DEFAULT_TABLE table=test-table [id=b2a7a4a3caa24c2a8db330ca16b79915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:24.045064 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef88d62e17bb4daca9de0818926d0b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:24.053285 6146 tablet_bootstrap.cc:492] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Bootstrap starting.
I20260810 19:24:24.057237 6147 tablet_bootstrap.cc:492] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Bootstrap starting.
I20260810 19:24:24.057823 6146 tablet_bootstrap.cc:654] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:24.059217 6148 tablet_bootstrap.cc:492] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Bootstrap starting.
I20260810 19:24:24.059388 6146 log.cc:824] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:24.061017 6147 tablet_bootstrap.cc:654] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:24.062271 6148 tablet_bootstrap.cc:654] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:24.062422 6147 log.cc:824] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:24.062861 6146 tablet_bootstrap.cc:492] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: No bootstrap required, opened a new log
I20260810 19:24:24.063182 6146 ts_tablet_manager.cc:1397] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 19:24:24.063251 6148 log.cc:824] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:24.066321 6148 tablet_bootstrap.cc:492] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: No bootstrap required, opened a new log
I20260810 19:24:24.066597 6148 ts_tablet_manager.cc:1397] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260810 19:24:24.066635 6147 tablet_bootstrap.cc:492] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: No bootstrap required, opened a new log
I20260810 19:24:24.066862 6147 ts_tablet_manager.cc:1397] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 19:24:24.073662 6147 raft_consensus.cc:348] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.074028 6147 raft_consensus.cc:374] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:24.074126 6147 raft_consensus.cc:729] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfddd73705045ee867d5d8870441dc3, State: Initialized, Role: FOLLOWER
I20260810 19:24:24.074090 6146 raft_consensus.cc:348] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.074563 6146 raft_consensus.cc:374] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:24.074667 6146 raft_consensus.cc:729] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e943e7796f43e98fec1d0e367756b0, State: Initialized, Role: FOLLOWER
I20260810 19:24:24.074959 6147 consensus_queue.cc:260] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.075367 6146 consensus_queue.cc:260] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.077203 5914 heartbeater.cc:503] Master 127.7.43.62:35595 was elected leader, sending a full tablet report...
I20260810 19:24:24.077407 6147 ts_tablet_manager.cc:1428] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260810 19:24:24.077585 6148 raft_consensus.cc:348] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.077937 6146 ts_tablet_manager.cc:1428] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260810 19:24:24.078073 6148 raft_consensus.cc:374] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:24.078182 6148 raft_consensus.cc:729] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06be987decb4d0fbc090f2ede80c718, State: Initialized, Role: FOLLOWER
I20260810 19:24:24.078781 6148 consensus_queue.cc:260] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.080307 6121 heartbeater.cc:503] Master 127.7.43.62:35595 was elected leader, sending a full tablet report...
I20260810 19:24:24.080544 6148 ts_tablet_manager.cc:1428] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260810 19:24:24.122202 5694 tablet.cc:2369] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:24:24.148921 5915 tablet.cc:2369] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:24:24.201452 6123 tablet.cc:2369] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:24:24.285292 6156 raft_consensus.cc:482] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:24.285640 6156 raft_consensus.cc:504] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.287372 6156 leader_election.cc:288] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06be987decb4d0fbc090f2ede80c718 (127.7.43.3:37719), 74e943e7796f43e98fec1d0e367756b0 (127.7.43.2:38787)
I20260810 19:24:24.294732 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.287931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:38977 (local address 127.7.43.3:37719)
0810 19:24:24.288163 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.288170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.289046 (+ 876us) server_negotiation.cc:407] Connection header received
0810 19:24:24.289187 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.289192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.289263 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.289364 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:24.289965 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.291113 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.292948 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.293791 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.293916 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.294015 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.294368 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.294491 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 19:24:24.294765 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.287941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:33975 (local address 127.7.43.2:38787)
0810 19:24:24.288192 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.288199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.288831 (+ 632us) server_negotiation.cc:407] Connection header received
0810 19:24:24.289034 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.289039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.289116 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.289212 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:24.289772 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.290840 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.292745 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.293713 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.293824 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.293921 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.294387 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.294510 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:24:24.294857 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.287785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:37719 (local address 127.7.43.1:38977)
0810 19:24:24.288918 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.288935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:24.289077 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:24.289388 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:24.289393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:24.289414 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:24.289833 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.289842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.291277 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.291280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.292772 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.292786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.294033 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.294038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.294196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.294230 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:24:24.294303 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 19:24:24.294423 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":34,"thread_start_us":672,"threads_started":1}
I20260810 19:24:24.295058 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.287679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:38787 (local address 127.7.43.1:33975)
0810 19:24:24.288723 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.288738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:24.288862 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:24.289221 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:24.289224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:24.289240 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:24.289633 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.289643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.290990 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.290995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.292592 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:24.292605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.294024 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.294033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:24.294209 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.294250 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 19:24:24.294337 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 19:24:24.294456 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":54,"thread_start_us":729,"threads_started":1}
I20260810 19:24:24.296032 6052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef88d62e17bb4daca9de0818926d0b2d" candidate_uuid: "4bfddd73705045ee867d5d8870441dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06be987decb4d0fbc090f2ede80c718" is_pre_election: true
I20260810 19:24:24.296322 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef88d62e17bb4daca9de0818926d0b2d" candidate_uuid: "4bfddd73705045ee867d5d8870441dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e943e7796f43e98fec1d0e367756b0" is_pre_election: true
I20260810 19:24:24.296536 6052 raft_consensus.cc:2454] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfddd73705045ee867d5d8870441dc3 in term 0.
I20260810 19:24:24.296792 5836 raft_consensus.cc:2454] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfddd73705045ee867d5d8870441dc3 in term 0.
I20260810 19:24:24.297171 5505 leader_election.cc:302] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfddd73705045ee867d5d8870441dc3, b06be987decb4d0fbc090f2ede80c718; no voters:
I20260810 19:24:24.297631 6156 raft_consensus.cc:2781] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:24:24.297768 6156 raft_consensus.cc:482] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:24:24.297865 6156 raft_consensus.cc:3037] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:24.300036 6156 raft_consensus.cc:504] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.300606 6156 leader_election.cc:288] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [CANDIDATE]: Term 1 election: Requested vote from peers b06be987decb4d0fbc090f2ede80c718 (127.7.43.3:37719), 74e943e7796f43e98fec1d0e367756b0 (127.7.43.2:38787)
I20260810 19:24:24.301131 6052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef88d62e17bb4daca9de0818926d0b2d" candidate_uuid: "4bfddd73705045ee867d5d8870441dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06be987decb4d0fbc090f2ede80c718"
I20260810 19:24:24.301136 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef88d62e17bb4daca9de0818926d0b2d" candidate_uuid: "4bfddd73705045ee867d5d8870441dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e943e7796f43e98fec1d0e367756b0"
I20260810 19:24:24.301340 5836 raft_consensus.cc:3037] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:24.301339 6052 raft_consensus.cc:3037] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:24.304522 5836 raft_consensus.cc:2454] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfddd73705045ee867d5d8870441dc3 in term 1.
I20260810 19:24:24.304540 6052 raft_consensus.cc:2454] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfddd73705045ee867d5d8870441dc3 in term 1.
I20260810 19:24:24.304920 5506 leader_election.cc:302] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfddd73705045ee867d5d8870441dc3, 74e943e7796f43e98fec1d0e367756b0; no voters:
I20260810 19:24:24.305186 6156 raft_consensus.cc:2781] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:24.305640 6156 raft_consensus.cc:686] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 1 LEADER]: Becoming Leader. State: Replica: 4bfddd73705045ee867d5d8870441dc3, State: Running, Role: LEADER
I20260810 19:24:24.306185 6156 consensus_queue.cc:237] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } }
I20260810 19:24:24.310573 5363 catalog_manager.cc:5495] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bfddd73705045ee867d5d8870441dc3 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "4bfddd73705045ee867d5d8870441dc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfddd73705045ee867d5d8870441dc3" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 } health_report { overall_health: UNKNOWN } } }
I20260810 19:24:24.332165 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.326975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.7.43.1:38909)
0810 19:24:24.327177 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.327182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.327206 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:24:24.327292 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.327295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.327374 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.327497 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:24:24.327841 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.328626 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.330049 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.330459 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.330568 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.331045 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:24:24.331562 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:24:24.331693 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.331833 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.331877 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:24:24.340562 6052 raft_consensus.cc:1261] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 1 FOLLOWER]: Refusing update from remote peer 4bfddd73705045ee867d5d8870441dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:24.340562 5836 raft_consensus.cc:1261] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 1 FOLLOWER]: Refusing update from remote peer 4bfddd73705045ee867d5d8870441dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:24.341737 6164 consensus_queue.cc:1035] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e943e7796f43e98fec1d0e367756b0" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:24.342093 6156 consensus_queue.cc:1035] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06be987decb4d0fbc090f2ede80c718" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:24.354090 6175 mvcc.cc:205] Tried to move back new op lower bound from 7317052884328177664 to 7317052884201943040. Current Snapshot: MvccSnapshot[applied={T|T < 7317052884328177664}]
I20260810 19:24:24.367554 6177 mvcc.cc:205] Tried to move back new op lower bound from 7317052884328177664 to 7317052884201943040. Current Snapshot: MvccSnapshot[applied={T|T < 7317052884328177664}]
I20260810 19:24:24.379177 5365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:24.379522 5365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:24.383715 5365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abbba351064e4a0d9b9dd4b8577fc120: Sending DeleteTablet for 3 replicas of tablet ef88d62e17bb4daca9de0818926d0b2d
I20260810 19:24:24.385147 6013 tablet_service.cc:1514] Processing DeleteTablet for tablet ef88d62e17bb4daca9de0818926d0b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260810 19:24:24.385107 5579 tablet_service.cc:1514] Processing DeleteTablet for tablet ef88d62e17bb4daca9de0818926d0b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:52686
I20260810 19:24:24.385202 6012 tablet_service.cc:1514] Processing DeleteTablet for tablet ef88d62e17bb4daca9de0818926d0b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260810 19:24:24.386600 5801 tablet_service.cc:1514] Processing DeleteTablet for tablet ef88d62e17bb4daca9de0818926d0b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:24 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:35486
I20260810 19:24:24.387563 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 5474
I20260810 19:24:24.417971 6190 tablet_replica.cc:330] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: stopping tablet replica
I20260810 19:24:24.418540 6190 raft_consensus.cc:2229] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:24.419142 6190 raft_consensus.cc:2258] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:24.423905 6195 tablet_replica.cc:330] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: stopping tablet replica
I20260810 19:24:24.424381 6195 raft_consensus.cc:2229] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:24.425289 6190 ts_tablet_manager.cc:1905] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:24:24 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:24:24.431845 6190 ts_tablet_manager.cc:1918] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:24.432019 6190 log.cc:1196] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/ef88d62e17bb4daca9de0818926d0b2d
I20260810 19:24:24.432540 6190 ts_tablet_manager.cc:1939] T ef88d62e17bb4daca9de0818926d0b2d P b06be987decb4d0fbc090f2ede80c718: Deleting consensus metadata
I20260810 19:24:24.432924 6195 raft_consensus.cc:2258] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:24.434338 6192 tablet_replica.cc:330] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: stopping tablet replica
I20260810 19:24:24.434551 5344 catalog_manager.cc:4841] TS b06be987decb4d0fbc090f2ede80c718 (127.7.43.3:37719): tablet ef88d62e17bb4daca9de0818926d0b2d (table test-table [id=b2a7a4a3caa24c2a8db330ca16b79915]) successfully deleted
I20260810 19:24:24.434818 6192 raft_consensus.cc:2229] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:24:24.435514 6192 raft_consensus.cc:2258] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:24.435707 6195 ts_tablet_manager.cc:1905] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 19:24:24.435773 5344 catalog_manager.cc:4808] TS b06be987decb4d0fbc090f2ede80c718 (127.7.43.3:37719): delete failed for tablet ef88d62e17bb4daca9de0818926d0b2d because the tablet was not found. No further retry: Not found: Tablet not found: ef88d62e17bb4daca9de0818926d0b2d
I20260810 19:24:24.437552 6192 ts_tablet_manager.cc:1905] T ef88d62e17bb4daca9de0818926d0b2d P 4bfddd73705045ee867d5d8870441dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:24.439230 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.388575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.7.43.1:38909)
0810 19:24:24.388748 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.388754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.414742 (+ 25988us) server_negotiation.cc:407] Connection header received
0810 19:24:24.415384 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.415391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.415491 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.415581 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:24.419651 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.420999 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.424727 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.425436 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.425571 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.431355 (+ 5784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:24.431374 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:24.431394 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:24.431447 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:24.435803 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:24.436710 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:24.436719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:24.436723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:24.436861 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:24.437245 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:24.437250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:24.437253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:24.437677 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:24.437987 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.438578 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.438852 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:24:24.440721 5681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:24.442579 6195 ts_tablet_manager.cc:1918] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:24.442762 6195 log.cc:1196] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/ef88d62e17bb4daca9de0818926d0b2d
I20260810 19:24:24.443332 6195 ts_tablet_manager.cc:1939] T ef88d62e17bb4daca9de0818926d0b2d P 74e943e7796f43e98fec1d0e367756b0: Deleting consensus metadata
I20260810 19:24:24.445070 5345 catalog_manager.cc:4841] TS 74e943e7796f43e98fec1d0e367756b0 (127.7.43.2:38787): tablet ef88d62e17bb4daca9de0818926d0b2d (table test-table [id=b2a7a4a3caa24c2a8db330ca16b79915]) successfully deleted
I20260810 19:24:24.540408 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 5474
I20260810 19:24:24.562999 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 5701
W20260810 19:24:24.563031 5343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:38909 (error 108)
I20260810 19:24:24.565083 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.564321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52726)
0810 19:24:24.564738 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.564874 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":196,"threads_started":1}
W20260810 19:24:24.565328 5343 catalog_manager.cc:4584] TS 4bfddd73705045ee867d5d8870441dc3 (127.7.43.1:38909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef88d62e17bb4daca9de0818926d0b2d: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Aug 10 19:24:24 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:24:24.577643 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.563819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.7.43.2:38787)
0810 19:24:24.564015 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.564020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.564168 (+ 148us) server_negotiation.cc:407] Connection header received
0810 19:24:24.564355 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.564360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.564451 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.564566 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:24.566106 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.566819 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.568095 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.568457 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.568550 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.572577 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:24.572591 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:24.572606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:24.572641 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:24.575317 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:24.575930 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:24.575937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:24.575939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:24.576082 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:24.576358 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:24.576361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:24.576363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:24.576721 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:24.576938 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.577164 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.577317 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:24:24.578264 5907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:24.618990 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.618492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52728)
0810 19:24:24.618741 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.618845 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 19:24:24.620052 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.619680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52730)
0810 19:24:24.619833 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.619912 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 19:24:24.666721 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 5701
I20260810 19:24:24.680387 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.679922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52734)
0810 19:24:24.680141 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.680246 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 19:24:24.681550 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.681154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52748)
0810 19:24:24.681300 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.681373 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 19:24:24.685266 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 5924
Aug 10 19:24:24 dist-test-slave-7lbf krb5kdc[5298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:24:24.699958 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.686043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.7.43.3:37719)
0810 19:24:24.686213 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:24.686218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:24.686237 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:24:24.686296 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:24.686300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:24.686361 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:24.686439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:24.687652 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.688344 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.689928 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:24.690341 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:24.690435 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:24.694916 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:24.694933 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:24.694950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:24.694984 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:24.697443 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:24.698206 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:24.698214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:24.698216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:24.698364 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:24.698694 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:24.698697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:24.698699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:24.699031 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:24.699243 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:24.699460 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 19:24:24.699638 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:24:24.700575 6110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:24.762435 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.761950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52760)
0810 19:24:24.762190 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.762283 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 19:24:24.763688 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:24.763208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:38909 (local address 127.0.0.1:52762)
0810 19:24:24.763458 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:24.763540 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 19:24:24.790526 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 5924
I20260810 19:24:24.816087 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 5319
I20260810 19:24:24.817340 5420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:24.926267 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 5319
2026-08-10T19:24:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4754 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T19:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:24Z Disabled control of system clock
I20260810 19:24:25.024176 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:33441
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:33441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:24:25.362527 6216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:25.362814 6216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:25.362905 6216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:25.371042 6216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:25.371155 6216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:25.371201 6216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:25.371258 6216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:25.371308 6216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:25.380597 6216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:33441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:33441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:25.382030 6216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:25.383895 6216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:25.396044 6224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:25.396090 6228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:25.397222 6226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:25.397464 6216 server_base.cc:1034] running on GCE node
I20260810 19:24:25.398536 6216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:25.400436 6216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:25.401667 6216 hybrid_clock.cc:648] HybridClock initialized: now 1786389865401605 us; error 57 us; skew 500 ppm
I20260810 19:24:25.402191 6216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:25.404680 6216 webserver.cc:460] Webserver started at http://127.7.43.62:33195/ using document root <none> and password file <none>
I20260810 19:24:25.405419 6216 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:25.405544 6216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:25.405877 6216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:25.408828 6216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "3cdbd4e0fe0e45c68496ae969ba3af16"
format_stamp: "Formatted at 2026-08-10 19:24:25 on dist-test-slave-7lbf"
server_key: "9d238276f7a274eec45acb66d389f247"
server_key_iv: "f4bd8280f785e6e0cdb34aace275e0aa"
server_key_version: "encryptionkey@0"
I20260810 19:24:25.409678 6216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "3cdbd4e0fe0e45c68496ae969ba3af16"
format_stamp: "Formatted at 2026-08-10 19:24:25 on dist-test-slave-7lbf"
server_key: "9d238276f7a274eec45acb66d389f247"
server_key_iv: "f4bd8280f785e6e0cdb34aace275e0aa"
server_key_version: "encryptionkey@0"
I20260810 19:24:25.415050 6216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 19:24:25.418599 6241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:25.419991 6216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:24:25.420156 6216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "3cdbd4e0fe0e45c68496ae969ba3af16"
format_stamp: "Formatted at 2026-08-10 19:24:25 on dist-test-slave-7lbf"
server_key: "9d238276f7a274eec45acb66d389f247"
server_key_iv: "f4bd8280f785e6e0cdb34aace275e0aa"
server_key_version: "encryptionkey@0"
I20260810 19:24:25.420380 6216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:25.437271 6216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:25.438218 6216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:25.438509 6216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:25.456270 6216 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:33441
I20260810 19:24:25.456292 6318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:33441 every 8 connection(s)
I20260810 19:24:25.457695 6216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:25.461447 6320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:25.463223 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 6216
I20260810 19:24:25.463406 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:25.464001 7340 external_mini_cluster.cc:1428] Setting key b709a85cdd885ec4ee70e14cf9a3d86d
I20260810 19:24:25.474687 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16: Bootstrap starting.
I20260810 19:24:25.477478 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:25.466056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.7.43.62:33441)
0810 19:24:25.467121 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:25.467133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:25.467178 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:24:25.468818 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:25.468847 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:25.469346 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:25.469792 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:25.470348 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:25.471676 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:25.473072 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:25.473815 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:25.473987 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:25.475228 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:25.475235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:25.475250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:25.475780 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:25.475864 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:25.475887 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:25.475993 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:25.476229 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 19:24:25.476525 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":405,"threads_started":1}
I20260810 19:24:25.478197 6320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:25.479404 6320 log.cc:824] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:25.482280 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16: No bootstrap required, opened a new log
I20260810 19:24:25.489691 6320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33441 } }
I20260810 19:24:25.490083 6320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:25.490187 6320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdbd4e0fe0e45c68496ae969ba3af16, State: Initialized, Role: FOLLOWER
I20260810 19:24:25.490850 6320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33441 } }
I20260810 19:24:25.491117 6320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:25.491256 6320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:25.491418 6320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:25.493625 6320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33441 } }
I20260810 19:24:25.494272 6320 leader_election.cc:302] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cdbd4e0fe0e45c68496ae969ba3af16; no voters:
I20260810 19:24:25.494954 6320 leader_election.cc:288] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:25.495054 6328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:25.496096 6328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [term 1 LEADER]: Becoming Leader. State: Replica: 3cdbd4e0fe0e45c68496ae969ba3af16, State: Running, Role: LEADER
I20260810 19:24:25.496614 6328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33441 } }
I20260810 19:24:25.497176 6320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:25.499478 6329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33441 } } }
I20260810 19:24:25.500023 6329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:25.499886 6330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cdbd4e0fe0e45c68496ae969ba3af16. Latest consensus state: current_term: 1 leader_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdbd4e0fe0e45c68496ae969ba3af16" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33441 } } }
I20260810 19:24:25.500185 6330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:25.502013 6342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:25.507253 6342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:25.516552 6342 catalog_manager.cc:1347] Generated new cluster ID: 5c06c69df0714174afd16b650c059976
I20260810 19:24:25.516661 6342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:25.532182 6342 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:25.533456 6342 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:25.546114 6342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16: Generated new TSK 0
I20260810 19:24:25.546990 6342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:25.565325 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:33441
--builtin_ntp_servers=127.7.43.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:24:25.924445 6368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:25.924748 6368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:25.924916 6368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:25.932404 6368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:25.932515 6368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:25.932657 6368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:24:25.941679 6368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:33441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:25.942966 6368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:25.944772 6368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:25.957707 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:25.957670 6381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:25.957746 6384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:25.958767 6368 server_base.cc:1034] running on GCE node
I20260810 19:24:25.959352 6368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:25.960271 6368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:25.961477 6368 hybrid_clock.cc:648] HybridClock initialized: now 1786389865961414 us; error 58 us; skew 500 ppm
I20260810 19:24:25.961814 6368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:25.963949 6368 webserver.cc:460] Webserver started at http://127.7.43.1:39313/ using document root <none> and password file <none>
I20260810 19:24:25.964660 6368 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:25.964768 6368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:25.965081 6368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:25.968071 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "9e2fd2007abe4c7c82519eed6bb06552"
format_stamp: "Formatted at 2026-08-10 19:24:25 on dist-test-slave-7lbf"
server_key: "2cf190697aa046afedffa4e50f8c7e3a"
server_key_iv: "76414e14cc7fb6c67391cc73effbed25"
server_key_version: "encryptionkey@0"
I20260810 19:24:25.968953 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "9e2fd2007abe4c7c82519eed6bb06552"
format_stamp: "Formatted at 2026-08-10 19:24:25 on dist-test-slave-7lbf"
server_key: "2cf190697aa046afedffa4e50f8c7e3a"
server_key_iv: "76414e14cc7fb6c67391cc73effbed25"
server_key_version: "encryptionkey@0"
I20260810 19:24:25.974646 6368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 19:24:25.978814 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:25.980299 6368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:25.980484 6368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "9e2fd2007abe4c7c82519eed6bb06552"
format_stamp: "Formatted at 2026-08-10 19:24:25 on dist-test-slave-7lbf"
server_key: "2cf190697aa046afedffa4e50f8c7e3a"
server_key_iv: "76414e14cc7fb6c67391cc73effbed25"
server_key_version: "encryptionkey@0"
I20260810 19:24:25.980715 6368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:26.017349 6368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:26.018329 6368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:26.018635 6368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:26.019748 6368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:26.021377 6368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:26.021483 6368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.021615 6368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:26.021679 6368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.052978 6368 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:40671
I20260810 19:24:26.053022 6591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:40671 every 8 connection(s)
I20260810 19:24:26.054339 6368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:24:26.060005 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 6368
I20260810 19:24:26.060148 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:24:26.060737 7340 external_mini_cluster.cc:1428] Setting key 06dbba43508a6c85c7d58ecf25a65410
I20260810 19:24:26.063882 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:33441
--builtin_ntp_servers=127.7.43.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:24:26.067965 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:26.056846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:53273 (local address 127.7.43.62:33441)
0810 19:24:26.057311 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:26.057318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:26.058608 (+ 1290us) server_negotiation.cc:407] Connection header received
0810 19:24:26.060473 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:26.060479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:26.060569 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:26.060700 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:26.061925 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.062942 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.065092 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.065500 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.065639 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:26.067116 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:26.067121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:26.067125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:26.067142 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:26.067182 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:26.067195 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:26.067272 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:26.067500 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 19:24:26.067603 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260810 19:24:26.069095 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:26.057497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:33441 (local address 127.7.43.1:53273)
0810 19:24:26.058441 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:26.058483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:26.060179 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:26.060909 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:26.060918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:26.061014 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:26.061718 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:26.061746 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.063114 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.063119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:26.064941 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:26.064959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.065755 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.065763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:26.066058 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:26.066781 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:26.066783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:26.066824 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:26.066919 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:26.067290 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:26.067303 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:26.067309 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:24:26.067473 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 19:24:26.067915 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":261,"threads_started":1}
I20260810 19:24:26.070643 6593 heartbeater.cc:347] Connected to a master server at 127.7.43.62:33441
I20260810 19:24:26.070964 6593 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:26.071725 6593 heartbeater.cc:511] Master 127.7.43.62:33441 requested a full tablet report, sending...
I20260810 19:24:26.073812 6268 ts_manager.cc:194] Registered new tserver with Master: 9e2fd2007abe4c7c82519eed6bb06552 (127.7.43.1:40671)
I20260810 19:24:26.075537 6268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.1:53273
W20260810 19:24:26.402691 6599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:26.402976 6599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:26.403110 6599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:26.410326 6599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:26.410440 6599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:26.410609 6599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:24:26.419832 6599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:33441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:26.421147 6599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:26.422971 6599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:26.435927 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:26.435963 6609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:26.436005 6611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:26.436967 6599 server_base.cc:1034] running on GCE node
I20260810 19:24:26.437605 6599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:26.438517 6599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:26.439721 6599 hybrid_clock.cc:648] HybridClock initialized: now 1786389866439668 us; error 46 us; skew 500 ppm
I20260810 19:24:26.440024 6599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:26.442245 6599 webserver.cc:460] Webserver started at http://127.7.43.2:46117/ using document root <none> and password file <none>
I20260810 19:24:26.442919 6599 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:26.443023 6599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:26.443338 6599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:26.446192 6599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "9f8f97baebd448c0ac4620b81d832dec"
format_stamp: "Formatted at 2026-08-10 19:24:26 on dist-test-slave-7lbf"
server_key: "de11438ab835c5cdd9f5a2a993c72af1"
server_key_iv: "6efb5ea1ea4bc3d8c045e3313d78f771"
server_key_version: "encryptionkey@0"
I20260810 19:24:26.446952 6599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "9f8f97baebd448c0ac4620b81d832dec"
format_stamp: "Formatted at 2026-08-10 19:24:26 on dist-test-slave-7lbf"
server_key: "de11438ab835c5cdd9f5a2a993c72af1"
server_key_iv: "6efb5ea1ea4bc3d8c045e3313d78f771"
server_key_version: "encryptionkey@0"
I20260810 19:24:26.451923 6599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:24:26.455343 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.456948 6599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:24:26.457122 6599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "9f8f97baebd448c0ac4620b81d832dec"
format_stamp: "Formatted at 2026-08-10 19:24:26 on dist-test-slave-7lbf"
server_key: "de11438ab835c5cdd9f5a2a993c72af1"
server_key_iv: "6efb5ea1ea4bc3d8c045e3313d78f771"
server_key_version: "encryptionkey@0"
I20260810 19:24:26.457363 6599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:26.481334 6599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:26.482156 6599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:26.482396 6599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:26.483338 6599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:26.484898 6599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:26.484987 6599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.485081 6599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:26.485153 6599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.515908 6599 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:46511
I20260810 19:24:26.515941 6802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:46511 every 8 connection(s)
I20260810 19:24:26.517189 6599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:24:26.527487 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 6599
I20260810 19:24:26.527673 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:24:26.528256 7340 external_mini_cluster.cc:1428] Setting key f43b69a0921fefe7f3df8883b9ed00db
I20260810 19:24:26.530225 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:26.519832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:38155 (local address 127.7.43.62:33441)
0810 19:24:26.520051 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:26.520057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:26.521734 (+ 1677us) server_negotiation.cc:407] Connection header received
0810 19:24:26.523608 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:26.523612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:26.523683 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:26.523767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:26.524938 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.525801 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.527327 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.527963 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.528087 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:26.529431 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:26.529437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:26.529440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:26.529457 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:26.529489 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:26.529501 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:26.529607 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:26.529813 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:24:26.529916 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:24:26.531032 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:33441
--builtin_ntp_servers=127.7.43.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:24:26.531327 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:26.520342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:33441 (local address 127.7.43.2:38155)
0810 19:24:26.521552 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:26.521611 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:26.523306 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:26.524046 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:26.524055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:26.524155 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:26.524775 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:26.524795 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.525942 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.525946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:26.527184 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:26.527196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.528185 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.528190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:26.528423 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:26.529066 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:26.529069 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:26.529110 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:26.529231 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:26.529603 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:26.529614 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:26.529623 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 19:24:26.529784 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 19:24:26.530228 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":407,"threads_started":1}
I20260810 19:24:26.532738 6803 heartbeater.cc:347] Connected to a master server at 127.7.43.62:33441
I20260810 19:24:26.533028 6803 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:26.533819 6803 heartbeater.cc:511] Master 127.7.43.62:33441 requested a full tablet report, sending...
I20260810 19:24:26.535235 6268 ts_manager.cc:194] Registered new tserver with Master: 9f8f97baebd448c0ac4620b81d832dec (127.7.43.2:46511)
I20260810 19:24:26.536213 6268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.2:38155
W20260810 19:24:26.880878 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:26.881161 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:26.881292 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:26.888547 6809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:26.888657 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:26.888800 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:24:26.897817 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:33441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:26.899084 6809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:26.900851 6809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:26.913754 6823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:26.913959 6817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:26.913813 6816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:26.914789 6809 server_base.cc:1034] running on GCE node
I20260810 19:24:26.915403 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:26.916301 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:26.917498 6809 hybrid_clock.cc:648] HybridClock initialized: now 1786389866917449 us; error 43 us; skew 500 ppm
I20260810 19:24:26.917841 6809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:26.920039 6809 webserver.cc:460] Webserver started at http://127.7.43.3:36505/ using document root <none> and password file <none>
I20260810 19:24:26.920702 6809 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:26.920807 6809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:26.921119 6809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:26.923951 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "ce26a5b2348c4b278d3cafa38ab7b16c"
format_stamp: "Formatted at 2026-08-10 19:24:26 on dist-test-slave-7lbf"
server_key: "83eb8a7a6472d0c4003e1957b5234330"
server_key_iv: "05860802f40d583a06bad68a0d64236b"
server_key_version: "encryptionkey@0"
I20260810 19:24:26.924772 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "ce26a5b2348c4b278d3cafa38ab7b16c"
format_stamp: "Formatted at 2026-08-10 19:24:26 on dist-test-slave-7lbf"
server_key: "83eb8a7a6472d0c4003e1957b5234330"
server_key_iv: "05860802f40d583a06bad68a0d64236b"
server_key_version: "encryptionkey@0"
I20260810 19:24:26.929914 6809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:26.933459 6832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.934767 6809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:26.934922 6809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "ce26a5b2348c4b278d3cafa38ab7b16c"
format_stamp: "Formatted at 2026-08-10 19:24:26 on dist-test-slave-7lbf"
server_key: "83eb8a7a6472d0c4003e1957b5234330"
server_key_iv: "05860802f40d583a06bad68a0d64236b"
server_key_version: "encryptionkey@0"
I20260810 19:24:26.935122 6809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:26.951514 6809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:26.952450 6809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:26.952764 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:26.953825 6809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:26.955461 6809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:26.955578 6809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.955696 6809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:26.955780 6809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:26.988735 6809 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:35585
I20260810 19:24:26.988781 7014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:35585 every 8 connection(s)
I20260810 19:24:26.990152 6809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:24:26.992812 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 6809
I20260810 19:24:26.992990 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:24:26.993640 7340 external_mini_cluster.cc:1428] Setting key a9c1a0504e58faee2a14337d9f09691a
I20260810 19:24:27.003126 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:26.992732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:53159 (local address 127.7.43.62:33441)
0810 19:24:26.992906 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:26.992911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:26.994724 (+ 1813us) server_negotiation.cc:407] Connection header received
0810 19:24:26.996968 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:26.996974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:26.997075 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:26.997160 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:26.998237 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.999106 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.000622 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.001062 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.001162 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.002441 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:27.002446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:27.002449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:27.002468 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:27.002501 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:27.002513 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:27.002585 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.002757 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.002836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:24:27.004083 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:26.993460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:33441 (local address 127.7.43.3:53159)
0810 19:24:26.994580 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:26.994623 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:26.996717 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:26.997345 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:26.997353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:26.997437 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:26.998055 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:26.998073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:26.999236 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:26.999240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.000475 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.000488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.001236 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.001241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.001437 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.002145 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:27.002148 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:27.002187 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:27.002263 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:27.002594 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:27.002600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:27.002606 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:24:27.002731 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 19:24:27.003037 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":330,"threads_started":1}
I20260810 19:24:27.005470 7015 heartbeater.cc:347] Connected to a master server at 127.7.43.62:33441
I20260810 19:24:27.005767 7015 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:27.006558 7015 heartbeater.cc:511] Master 127.7.43.62:33441 requested a full tablet report, sending...
I20260810 19:24:27.008072 6268 ts_manager.cc:194] Registered new tserver with Master: ce26a5b2348c4b278d3cafa38ab7b16c (127.7.43.3:35585)
I20260810 19:24:27.009019 6268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.3:53159
I20260810 19:24:27.009125 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:24:27.018950 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.013179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.7.43.62:33441)
0810 19:24:27.013327 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.013332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.013806 (+ 474us) server_negotiation.cc:407] Connection header received
0810 19:24:27.013997 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.014001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.014071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.014148 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:27.014571 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.015235 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.016763 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.017194 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.017285 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.018310 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:27.018314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:27.018318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:27.018335 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:27.018365 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:27.018377 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:27.018483 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.018604 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.018647 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:24:27.022598 6268 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:24:27.029642 6268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:24:27.053215 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.041885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:46511 (local address 127.0.0.1:49098)
0810 19:24:27.043105 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:27.043140 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:27.043312 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:27.044309 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:27.044318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:27.044349 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:27.044787 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.044803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.046944 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.046950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.049446 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.049464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.052088 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.052095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.052326 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.052393 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 19:24:27.052643 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 19:24:27.052859 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":761,"threads_started":1}
I20260810 19:24:27.054262 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.041909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.7.43.2:46511)
0810 19:24:27.042486 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.042495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.043247 (+ 752us) server_negotiation.cc:407] Connection header received
0810 19:24:27.043554 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.043564 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.043833 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.044041 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:27.045028 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.046764 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.049704 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.051508 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.051625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.051889 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.053688 (+ 1799us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.053890 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":305,"threads_started":1}
I20260810 19:24:27.065006 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.044014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:35585 (local address 127.0.0.1:43116)
0810 19:24:27.054987 (+ 10973us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:27.055008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:27.055117 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:27.058721 (+ 3604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:27.058726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:27.058753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:27.058930 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.058938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.060422 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.060427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.062144 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.062161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.064221 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.064228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.064442 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.064475 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:27.064567 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 19:24:27.064700 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10850,"thread_start_us":16378,"threads_started":1}
I20260810 19:24:27.064603 6696 tablet_service.cc:1467] Processing CreateTablet for tablet 1decda0e80c245dc91dadd6dc96cbb06 (DEFAULT_TABLE table=test-table [id=85e544195b284b5e948d0bc55ebe82c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:27.065780 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.044583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.7.43.3:35585)
0810 19:24:27.058055 (+ 13472us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.058064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.058103 (+ 39us) server_negotiation.cc:407] Connection header received
0810 19:24:27.058202 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.058210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.058500 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.059074 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:27.059284 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.060256 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.062325 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.063702 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.063833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.064114 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.065236 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.065404 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13302,"thread_start_us":13143,"threads_started":1}
I20260810 19:24:27.066256 6696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1decda0e80c245dc91dadd6dc96cbb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:27.067544 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.043448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40671 (local address 127.0.0.1:60460)
0810 19:24:27.043970 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:27.043987 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:27.044089 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:27.061381 (+ 17292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:27.061386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:27.061415 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:27.061878 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.061892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.063436 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.063442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.065255 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.065271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.066832 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.066839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.067020 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.067053 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:27.067139 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:24:27.067262 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":228,"threads_started":1}
I20260810 19:24:27.067683 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.043770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.7.43.1:40671)
0810 19:24:27.060814 (+ 17044us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.060822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.060851 (+ 29us) server_negotiation.cc:407] Connection header received
0810 19:24:27.060938 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.060945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.061174 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.061372 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:27.062082 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.063310 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.065437 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.066648 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.066770 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.067040 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.067185 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.067337 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16866,"thread_start_us":16755,"threads_started":1}
I20260810 19:24:27.069489 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 1decda0e80c245dc91dadd6dc96cbb06 (DEFAULT_TABLE table=test-table [id=85e544195b284b5e948d0bc55ebe82c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:27.070972 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1decda0e80c245dc91dadd6dc96cbb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:27.070621 6481 tablet_service.cc:1467] Processing CreateTablet for tablet 1decda0e80c245dc91dadd6dc96cbb06 (DEFAULT_TABLE table=test-table [id=85e544195b284b5e948d0bc55ebe82c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:27.071990 6481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1decda0e80c245dc91dadd6dc96cbb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:27.079530 6593 heartbeater.cc:503] Master 127.7.43.62:33441 was elected leader, sending a full tablet report...
I20260810 19:24:27.082482 7054 tablet_bootstrap.cc:492] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Bootstrap starting.
I20260810 19:24:27.083308 7057 tablet_bootstrap.cc:492] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Bootstrap starting.
I20260810 19:24:27.086189 7054 tablet_bootstrap.cc:654] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:27.087193 7057 tablet_bootstrap.cc:654] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:27.087208 7054 log.cc:824] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:27.088404 7057 log.cc:824] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:27.088548 7058 tablet_bootstrap.cc:492] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Bootstrap starting.
I20260810 19:24:27.090353 7054 tablet_bootstrap.cc:492] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: No bootstrap required, opened a new log
I20260810 19:24:27.090652 7054 ts_tablet_manager.cc:1397] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260810 19:24:27.091456 7057 tablet_bootstrap.cc:492] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: No bootstrap required, opened a new log
I20260810 19:24:27.092104 7057 ts_tablet_manager.cc:1397] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 19:24:27.092290 7058 tablet_bootstrap.cc:654] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:27.093449 7058 log.cc:824] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:27.096493 7058 tablet_bootstrap.cc:492] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: No bootstrap required, opened a new log
I20260810 19:24:27.096762 7058 ts_tablet_manager.cc:1397] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:24:27.099380 7057 raft_consensus.cc:348] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.099737 7057 raft_consensus.cc:374] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:27.099838 7057 raft_consensus.cc:729] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2fd2007abe4c7c82519eed6bb06552, State: Initialized, Role: FOLLOWER
I20260810 19:24:27.100387 7057 consensus_queue.cc:260] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.100539 7054 raft_consensus.cc:348] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.101038 7054 raft_consensus.cc:374] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:27.101174 7054 raft_consensus.cc:729] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8f97baebd448c0ac4620b81d832dec, State: Initialized, Role: FOLLOWER
I20260810 19:24:27.102047 7057 ts_tablet_manager.cc:1428] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260810 19:24:27.101887 7054 consensus_queue.cc:260] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.106369 6803 heartbeater.cc:503] Master 127.7.43.62:33441 was elected leader, sending a full tablet report...
I20260810 19:24:27.106843 7054 ts_tablet_manager.cc:1428] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260810 19:24:27.107213 7058 raft_consensus.cc:348] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.107653 7058 raft_consensus.cc:374] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:27.107764 7058 raft_consensus.cc:729] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce26a5b2348c4b278d3cafa38ab7b16c, State: Initialized, Role: FOLLOWER
I20260810 19:24:27.108409 7058 consensus_queue.cc:260] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.109503 7015 heartbeater.cc:503] Master 127.7.43.62:33441 was elected leader, sending a full tablet report...
I20260810 19:24:27.110004 7058 ts_tablet_manager.cc:1428] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260810 19:24:27.137005 7064 raft_consensus.cc:482] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:27.137329 7064 raft_consensus.cc:504] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.139192 7064 leader_election.cc:288] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f8f97baebd448c0ac4620b81d832dec (127.7.43.2:46511), ce26a5b2348c4b278d3cafa38ab7b16c (127.7.43.3:35585)
I20260810 19:24:27.146986 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.139636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:49173 (local address 127.7.43.2:46511)
0810 19:24:27.139851 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.139858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.140213 (+ 355us) server_negotiation.cc:407] Connection header received
0810 19:24:27.140519 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.140525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.140621 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.140689 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:27.141539 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.142489 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.145004 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.146183 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.146317 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.146412 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.146626 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.146762 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 19:24:27.147086 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.139437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:46511 (local address 127.7.43.1:49173)
0810 19:24:27.140093 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:27.140121 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:27.140346 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:27.140910 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:27.140915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:27.140944 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:27.141381 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.141393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.142623 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.142627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.144794 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.144810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.146350 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.146354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.146467 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.146492 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:24:27.146579 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 19:24:27.146712 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":16,"thread_start_us":323,"threads_started":1}
I20260810 19:24:27.147331 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.140446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:39275 (local address 127.7.43.3:35585)
0810 19:24:27.140655 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.140660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.141285 (+ 625us) server_negotiation.cc:407] Connection header received
0810 19:24:27.141447 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.141452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.141557 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.141657 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:27.142083 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.142954 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.145247 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.146475 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.146576 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.146670 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.146984 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.147112 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 19:24:27.147538 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.140215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:35585 (local address 127.7.43.1:39275)
0810 19:24:27.141172 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:27.141188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:27.141319 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:27.141670 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:27.141673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:27.141693 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:27.141942 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.141949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.143096 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.143101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.145074 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:27.145085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.146655 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.146661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:27.146820 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.146853 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:27.146939 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:24:27.147059 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":34,"thread_start_us":614,"threads_started":1}
I20260810 19:24:27.148339 6731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1decda0e80c245dc91dadd6dc96cbb06" candidate_uuid: "9e2fd2007abe4c7c82519eed6bb06552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8f97baebd448c0ac4620b81d832dec" is_pre_election: true
I20260810 19:24:27.148464 6943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1decda0e80c245dc91dadd6dc96cbb06" candidate_uuid: "9e2fd2007abe4c7c82519eed6bb06552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" is_pre_election: true
I20260810 19:24:27.148887 6731 raft_consensus.cc:2454] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2fd2007abe4c7c82519eed6bb06552 in term 0.
I20260810 19:24:27.148955 6943 raft_consensus.cc:2454] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2fd2007abe4c7c82519eed6bb06552 in term 0.
I20260810 19:24:27.149582 6401 leader_election.cc:302] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2fd2007abe4c7c82519eed6bb06552, 9f8f97baebd448c0ac4620b81d832dec; no voters:
I20260810 19:24:27.149955 7064 raft_consensus.cc:2781] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:24:27.150115 7064 raft_consensus.cc:482] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:24:27.150240 7064 raft_consensus.cc:3037] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:27.152520 7064 raft_consensus.cc:504] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.153152 7064 leader_election.cc:288] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [CANDIDATE]: Term 1 election: Requested vote from peers 9f8f97baebd448c0ac4620b81d832dec (127.7.43.2:46511), ce26a5b2348c4b278d3cafa38ab7b16c (127.7.43.3:35585)
I20260810 19:24:27.153545 6731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1decda0e80c245dc91dadd6dc96cbb06" candidate_uuid: "9e2fd2007abe4c7c82519eed6bb06552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8f97baebd448c0ac4620b81d832dec"
I20260810 19:24:27.153644 6943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1decda0e80c245dc91dadd6dc96cbb06" candidate_uuid: "9e2fd2007abe4c7c82519eed6bb06552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c"
I20260810 19:24:27.153754 6731 raft_consensus.cc:3037] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:27.153844 6943 raft_consensus.cc:3037] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:27.156002 6731 raft_consensus.cc:2454] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2fd2007abe4c7c82519eed6bb06552 in term 1.
I20260810 19:24:27.156227 6943 raft_consensus.cc:2454] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2fd2007abe4c7c82519eed6bb06552 in term 1.
I20260810 19:24:27.156392 6401 leader_election.cc:302] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2fd2007abe4c7c82519eed6bb06552, 9f8f97baebd448c0ac4620b81d832dec; no voters:
I20260810 19:24:27.156665 7064 raft_consensus.cc:2781] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:27.157088 7064 raft_consensus.cc:686] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2fd2007abe4c7c82519eed6bb06552, State: Running, Role: LEADER
I20260810 19:24:27.157631 7064 consensus_queue.cc:237] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } }
I20260810 19:24:27.162060 6268 catalog_manager.cc:5495] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2fd2007abe4c7c82519eed6bb06552 (127.7.43.1). New cstate: current_term: 1 leader_uuid: "9e2fd2007abe4c7c82519eed6bb06552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e2fd2007abe4c7c82519eed6bb06552" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20260810 19:24:27.207427 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.202077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.7.43.1:40671)
0810 19:24:27.202304 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.202309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.202324 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:24:27.202410 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.202414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.202503 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.202663 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:24:27.203042 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.203748 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.205118 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.205578 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.205688 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.206278 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:24:27.206801 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:24:27.206983 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.207166 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.207212 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:24:27.215945 6943 raft_consensus.cc:1261] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 1 FOLLOWER]: Refusing update from remote peer 9e2fd2007abe4c7c82519eed6bb06552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:27.215939 6731 raft_consensus.cc:1261] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 1 FOLLOWER]: Refusing update from remote peer 9e2fd2007abe4c7c82519eed6bb06552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:27.216887 7069 consensus_queue.cc:1035] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce26a5b2348c4b278d3cafa38ab7b16c" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:27.217283 7064 consensus_queue.cc:1035] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f8f97baebd448c0ac4620b81d832dec" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:27.242897 6267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:27.243170 6267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:27.246587 6267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cdbd4e0fe0e45c68496ae969ba3af16: Sending DeleteTablet for 3 replicas of tablet 1decda0e80c245dc91dadd6dc96cbb06
I20260810 19:24:27.247718 6696 tablet_service.cc:1514] Processing DeleteTablet for tablet 1decda0e80c245dc91dadd6dc96cbb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:27 UTC) from {username='slave'} at 127.0.0.1:49098
I20260810 19:24:27.248097 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 1decda0e80c245dc91dadd6dc96cbb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:27 UTC) from {username='slave'} at 127.0.0.1:43116
I20260810 19:24:27.248107 6481 tablet_service.cc:1514] Processing DeleteTablet for tablet 1decda0e80c245dc91dadd6dc96cbb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:27 UTC) from {username='slave'} at 127.0.0.1:60460
I20260810 19:24:27.249588 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 6368
I20260810 19:24:27.258601 7091 tablet_replica.cc:330] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: stopping tablet replica
I20260810 19:24:27.259117 7091 raft_consensus.cc:2229] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:24:27.259886 7091 raft_consensus.cc:2258] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:27.260711 7090 tablet_replica.cc:330] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: stopping tablet replica
I20260810 19:24:27.261194 7090 raft_consensus.cc:2229] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:27.261847 7090 raft_consensus.cc:2258] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:27.262367 7088 tablet_replica.cc:330] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: stopping tablet replica
I20260810 19:24:27.263586 7088 raft_consensus.cc:2229] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:27.264254 7088 raft_consensus.cc:2258] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:27.264274 7090 ts_tablet_manager.cc:1905] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:27.265758 7091 ts_tablet_manager.cc:1905] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:27.268052 7088 ts_tablet_manager.cc:1905] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:27.274282 7091 ts_tablet_manager.cc:1918] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:27.274428 7090 ts_tablet_manager.cc:1918] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:27.274683 7090 log.cc:1196] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/1decda0e80c245dc91dadd6dc96cbb06
I20260810 19:24:27.274832 7091 log.cc:1196] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/1decda0e80c245dc91dadd6dc96cbb06
I20260810 19:24:27.275295 7090 ts_tablet_manager.cc:1939] T 1decda0e80c245dc91dadd6dc96cbb06 P ce26a5b2348c4b278d3cafa38ab7b16c: Deleting consensus metadata
I20260810 19:24:27.275568 7091 ts_tablet_manager.cc:1939] T 1decda0e80c245dc91dadd6dc96cbb06 P 9e2fd2007abe4c7c82519eed6bb06552: Deleting consensus metadata
I20260810 19:24:27.276922 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.251369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.7.43.1:40671)
0810 19:24:27.251646 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.251652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.264884 (+ 13232us) server_negotiation.cc:407] Connection header received
0810 19:24:27.265395 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.265401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.265495 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.265607 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:27.266302 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.267523 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.269446 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.270126 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.270228 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.275038 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:27.275055 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:27.275073 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:27.275961 (+ 888us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:27.276037 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:27.276063 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:27.276143 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.276428 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.276540 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":150}
I20260810 19:24:27.277335 6246 catalog_manager.cc:4841] TS ce26a5b2348c4b278d3cafa38ab7b16c (127.7.43.3:35585): tablet 1decda0e80c245dc91dadd6dc96cbb06 (table test-table [id=85e544195b284b5e948d0bc55ebe82c5]) successfully deleted
I20260810 19:24:27.277738 6246 catalog_manager.cc:4841] TS 9e2fd2007abe4c7c82519eed6bb06552 (127.7.43.1:40671): tablet 1decda0e80c245dc91dadd6dc96cbb06 (table test-table [id=85e544195b284b5e948d0bc55ebe82c5]) successfully deleted
I20260810 19:24:27.278084 6584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:27.280774 7088 ts_tablet_manager.cc:1918] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:27.280963 7088 log.cc:1196] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/1decda0e80c245dc91dadd6dc96cbb06
I20260810 19:24:27.281481 7088 ts_tablet_manager.cc:1939] T 1decda0e80c245dc91dadd6dc96cbb06 P 9f8f97baebd448c0ac4620b81d832dec: Deleting consensus metadata
I20260810 19:24:27.282972 6246 catalog_manager.cc:4841] TS 9f8f97baebd448c0ac4620b81d832dec (127.7.43.2:46511): tablet 1decda0e80c245dc91dadd6dc96cbb06 (table test-table [id=85e544195b284b5e948d0bc55ebe82c5]) successfully deleted
I20260810 19:24:27.373874 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 6368
I20260810 19:24:27.401928 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 6599
I20260810 19:24:27.408584 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.402800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.7.43.2:46511)
0810 19:24:27.402986 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.402991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.403010 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:24:27.403137 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.403141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.403263 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.403360 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:27.403745 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.404560 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.405865 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.406270 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.406366 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.407333 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:27.407342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:27.407356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:27.407929 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:27.407980 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:27.408001 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:27.408085 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.408166 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.408253 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:24:27.409248 6792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:27.495900 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 6599
I20260810 19:24:27.522403 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 6809
I20260810 19:24:27.529245 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:27.523220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.7.43.3:35585)
0810 19:24:27.523410 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:27.523416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:27.523437 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:24:27.523643 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:27.523647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:27.523748 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:27.523835 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:27.524248 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.525006 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.526371 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:27.526769 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:27.526881 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:27.527948 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:27.527958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:27.527975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:27.528534 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:27.528588 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:27.528608 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:27.528706 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:27.528841 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 19:24:27.528950 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:24:27.530007 7003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:27.615911 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 6809
I20260810 19:24:27.634871 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 6216
I20260810 19:24:27.635969 6314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:27.743692 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 6216
2026-08-10T19:24:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2820 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:27 dist-test-slave-7lbf krb5kdc[7103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:27 dist-test-slave-7lbf krb5kdc[7103](info): set up 2 sockets
Aug 10 19:24:27 dist-test-slave-7lbf krb5kdc[7103](info): commencing operation
krb5kdc: starting...
W20260810 19:24:29.010308 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:29 dist-test-slave-7lbf krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:24:29.326920 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39265
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:29.670571 7124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:29.670886 7124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:29.671025 7124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:29.678439 7124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:29.678560 7124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:29.678637 7124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:29.678717 7124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:29.678779 7124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:29.688830 7124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:29.690284 7124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:29.691672 7124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:29.703776 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:29.703732 7132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:29.703852 7142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:29.705298 7124 server_base.cc:1034] running on GCE node
I20260810 19:24:29.706122 7124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:29.707587 7124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:29.708796 7124 hybrid_clock.cc:648] HybridClock initialized: now 1786389869708741 us; error 49 us; skew 500 ppm
I20260810 19:24:29.709141 7124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:29 dist-test-slave-7lbf krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389869, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:29.712220 7124 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:24:29.713820 7124 webserver.cc:460] Webserver started at http://127.7.43.62:37797/ using document root <none> and password file <none>
I20260810 19:24:29.714495 7124 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:29.714601 7124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:29.714862 7124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:29.718006 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "78d56ac65420424e971b63968b0b1416"
format_stamp: "Formatted at 2026-08-10 19:24:29 on dist-test-slave-7lbf"
server_key: "af439a6c197fed902cd633f6c3e0dbad"
server_key_iv: "b048f02378a6c7e5ed5e3035f9f936b4"
server_key_version: "encryptionkey@0"
I20260810 19:24:29.718732 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "78d56ac65420424e971b63968b0b1416"
format_stamp: "Formatted at 2026-08-10 19:24:29 on dist-test-slave-7lbf"
server_key: "af439a6c197fed902cd633f6c3e0dbad"
server_key_iv: "b048f02378a6c7e5ed5e3035f9f936b4"
server_key_version: "encryptionkey@0"
I20260810 19:24:29.724126 7124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:24:29.727725 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:29.729193 7124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:24:29.729349 7124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "78d56ac65420424e971b63968b0b1416"
format_stamp: "Formatted at 2026-08-10 19:24:29 on dist-test-slave-7lbf"
server_key: "af439a6c197fed902cd633f6c3e0dbad"
server_key_iv: "b048f02378a6c7e5ed5e3035f9f936b4"
server_key_version: "encryptionkey@0"
I20260810 19:24:29.729612 7124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:29.753357 7124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:29.757752 7124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:29.758044 7124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:29.776234 7124 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:39265
I20260810 19:24:29.776268 7238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:39265 every 8 connection(s)
I20260810 19:24:29.777673 7124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:29.781483 7240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:29.787951 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7124
I20260810 19:24:29.788156 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:29.788759 7340 external_mini_cluster.cc:1428] Setting key 8569b0463355c7ba06fc19dce9caf187
I20260810 19:24:29.792724 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416: Bootstrap starting.
I20260810 19:24:29.796593 7240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:29.797748 7240 log.cc:824] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:29.801000 7240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416: No bootstrap required, opened a new log
Aug 10 19:24:29 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:29.809592 7240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d56ac65420424e971b63968b0b1416" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39265 } }
I20260810 19:24:29.810021 7240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:29.810125 7240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d56ac65420424e971b63968b0b1416, State: Initialized, Role: FOLLOWER
I20260810 19:24:29.810885 7240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d56ac65420424e971b63968b0b1416" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39265 } }
I20260810 19:24:29.811174 7240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:29.811319 7240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:29.811510 7240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:29.812546 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:29.790802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.7.43.62:39265)
0810 19:24:29.791588 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:29.791600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:29.791641 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:24:29.793401 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:29.793425 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:29.793871 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:29.794214 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:29.796097 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:29.797472 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:29.799125 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:29.799793 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:29.799941 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:29.804660 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:29.804669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:29.804685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:29.804728 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:29.807866 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:29.808571 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:29.808578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:29.808588 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:29.808768 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:29.809157 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:29.809161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:29.809163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:29.809748 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:29.810062 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:29.810576 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 19:24:29.811030 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":363,"threads_started":1}
I20260810 19:24:29.814350 7240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d56ac65420424e971b63968b0b1416" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39265 } }
I20260810 19:24:29.814865 7240 leader_election.cc:302] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d56ac65420424e971b63968b0b1416; no voters:
I20260810 19:24:29.815480 7240 leader_election.cc:288] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:29.815603 7251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:29.816335 7251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [term 1 LEADER]: Becoming Leader. State: Replica: 78d56ac65420424e971b63968b0b1416, State: Running, Role: LEADER
I20260810 19:24:29.816880 7251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d56ac65420424e971b63968b0b1416" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39265 } }
I20260810 19:24:29.817571 7240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:29.819577 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d56ac65420424e971b63968b0b1416. Latest consensus state: current_term: 1 leader_uuid: "78d56ac65420424e971b63968b0b1416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d56ac65420424e971b63968b0b1416" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39265 } } }
I20260810 19:24:29.820005 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:29.820155 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d56ac65420424e971b63968b0b1416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d56ac65420424e971b63968b0b1416" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39265 } } }
I20260810 19:24:29.820510 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:29.821131 7264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:29.825927 7264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:29.835530 7264 catalog_manager.cc:1347] Generated new cluster ID: 2776ec9ad9584ccb9edb79e6d14a871a
I20260810 19:24:29.835654 7264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:29.847918 7264 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:29.849247 7264 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:29.858088 7264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416: Generated new TSK 0
I20260810 19:24:29.858918 7264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.43.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
Entry for principal kudu/127.7.43.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab.
I20260810 19:24:29.935055 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39265
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:30.274499 7284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:30.274796 7284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:30.274932 7284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:30.282181 7284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:30.282294 7284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:30.282444 7284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:24:30.292164 7284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.1.keytab
--principal=kudu/127.7.43.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:30.293479 7284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:30.294956 7284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:30.308004 7297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:30.308086 7295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:30.308063 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:30.308801 7284 server_base.cc:1034] running on GCE node
I20260810 19:24:30.309401 7284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:30.310359 7284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:30.311563 7284 hybrid_clock.cc:648] HybridClock initialized: now 1786389870311511 us; error 39 us; skew 500 ppm
I20260810 19:24:30.311865 7284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:30 dist-test-slave-7lbf krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389870, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:30.314960 7284 init.cc:377] Logged in from keytab as kudu/127.7.43.1@KRBTEST.COM (short username kudu)
I20260810 19:24:30.316342 7284 webserver.cc:460] Webserver started at http://127.7.43.1:44227/ using document root <none> and password file <none>
I20260810 19:24:30.317034 7284 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:30.317150 7284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:30.317468 7284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:30.320539 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "85ce1a120c44412abb39de9fec8c4b1a"
format_stamp: "Formatted at 2026-08-10 19:24:30 on dist-test-slave-7lbf"
server_key: "9d64835812798fbd615d0529eea5efd7"
server_key_iv: "b3450eac5146178a1cdc0cc167b920e1"
server_key_version: "encryptionkey@0"
I20260810 19:24:30.321314 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "85ce1a120c44412abb39de9fec8c4b1a"
format_stamp: "Formatted at 2026-08-10 19:24:30 on dist-test-slave-7lbf"
server_key: "9d64835812798fbd615d0529eea5efd7"
server_key_iv: "b3450eac5146178a1cdc0cc167b920e1"
server_key_version: "encryptionkey@0"
I20260810 19:24:30.326581 7284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:24:30.330101 7313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:30.331494 7284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:24:30.331645 7284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "85ce1a120c44412abb39de9fec8c4b1a"
format_stamp: "Formatted at 2026-08-10 19:24:30 on dist-test-slave-7lbf"
server_key: "9d64835812798fbd615d0529eea5efd7"
server_key_iv: "b3450eac5146178a1cdc0cc167b920e1"
server_key_version: "encryptionkey@0"
I20260810 19:24:30.331840 7284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:30.354063 7284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:30.358327 7284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:30.358613 7284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:30.359575 7284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:30.360950 7284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:30.361043 7284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:30.361162 7284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:30.361215 7284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:30.393263 7284 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:40753
I20260810 19:24:30.393311 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:40753 every 8 connection(s)
I20260810 19:24:30.394610 7284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:24:30.396674 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7284
I20260810 19:24:30.396832 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:24:30.397372 7340 external_mini_cluster.cc:1428] Setting key b74ea9723853a5974b772f03c48fc5fd
Aug 10 19:24:30 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389870, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.1@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:30.417027 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:30.397208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:59525 (local address 127.7.43.62:39265)
0810 19:24:30.397734 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:30.397741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:30.398785 (+ 1044us) server_negotiation.cc:407] Connection header received
0810 19:24:30.400843 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:30.400849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:30.400939 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:30.401038 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:30.402668 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.403674 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.406106 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.406738 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.406869 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:30.411060 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:30.411065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:30.411069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:30.411103 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:30.413727 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:30.414677 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:30.414682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:30.414686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:30.414841 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:30.415360 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:30.415363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:30.415366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:30.415634 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:30.415749 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:30.416575 (+ 826us) server_negotiation.cc:299] Negotiation successful
0810 19:24:30.416755 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260810 19:24:30.418637 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:30.397803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39265 (local address 127.7.43.1:59525)
0810 19:24:30.398618 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:30.398667 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:30.400586 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:30.401290 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:30.401298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:30.401900 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:30.402479 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:30.402499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.403883 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.403889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:30.405901 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:30.405916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.406965 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.406972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:30.407268 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:30.408007 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:30.408060 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:30.410812 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:30.413886 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:30.413896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:30.413916 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:30.414519 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:30.414976 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:30.414981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:30.414984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:30.415247 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:30.415759 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:30.415767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:30.416201 (+ 434us) client_negotiation.cc:769] Sending connection context
0810 19:24:30.416491 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 19:24:30.417044 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":327,"threads_started":1}
I20260810 19:24:30.420207 7526 heartbeater.cc:347] Connected to a master server at 127.7.43.62:39265
I20260810 19:24:30.420542 7526 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:30.421444 7526 heartbeater.cc:511] Master 127.7.43.62:39265 requested a full tablet report, sending...
I20260810 19:24:30.423771 7183 ts_manager.cc:194] Registered new tserver with Master: 85ce1a120c44412abb39de9fec8c4b1a (127.7.43.1:40753)
I20260810 19:24:30.425216 7183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.1@KRBTEST.COM'} at 127.7.43.1:59525
WARNING: no policy specified for kudu/127.7.43.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
Entry for principal kudu/127.7.43.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab.
I20260810 19:24:30.470130 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39265
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:30.809706 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:30.809993 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:30.810097 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:30.817219 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:30.817337 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:30.817462 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:24:30.827122 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.2.keytab
--principal=kudu/127.7.43.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:30.828398 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:30.829857 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:30.842988 7548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:30.842988 7549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:30.842968 7551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:30.843981 7533 server_base.cc:1034] running on GCE node
I20260810 19:24:30.844589 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:30.845539 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:30.846735 7533 hybrid_clock.cc:648] HybridClock initialized: now 1786389870846678 us; error 50 us; skew 500 ppm
I20260810 19:24:30.847052 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:30 dist-test-slave-7lbf krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389870, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:30.850240 7533 init.cc:377] Logged in from keytab as kudu/127.7.43.2@KRBTEST.COM (short username kudu)
I20260810 19:24:30.851718 7533 webserver.cc:460] Webserver started at http://127.7.43.2:38937/ using document root <none> and password file <none>
I20260810 19:24:30.852412 7533 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:30.852531 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:30.852824 7533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:30.855798 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "ababb62988d34cb9b0a12bf8ac90de28"
format_stamp: "Formatted at 2026-08-10 19:24:30 on dist-test-slave-7lbf"
server_key: "1bf9344bd185d481f573250ebeb0e37a"
server_key_iv: "a9a5610a69997cf16d19d199b8563de0"
server_key_version: "encryptionkey@0"
I20260810 19:24:30.856578 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "ababb62988d34cb9b0a12bf8ac90de28"
format_stamp: "Formatted at 2026-08-10 19:24:30 on dist-test-slave-7lbf"
server_key: "1bf9344bd185d481f573250ebeb0e37a"
server_key_iv: "a9a5610a69997cf16d19d199b8563de0"
server_key_version: "encryptionkey@0"
I20260810 19:24:30.861897 7533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 19:24:30.865267 7562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:30.866617 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:30.866775 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "ababb62988d34cb9b0a12bf8ac90de28"
format_stamp: "Formatted at 2026-08-10 19:24:30 on dist-test-slave-7lbf"
server_key: "1bf9344bd185d481f573250ebeb0e37a"
server_key_iv: "a9a5610a69997cf16d19d199b8563de0"
server_key_version: "encryptionkey@0"
I20260810 19:24:30.866959 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:30.888892 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:30.893450 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:30.893757 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:30.894801 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:30.896198 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:30.896282 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:30.896401 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:30.896456 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:30.927222 7533 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:45903
I20260810 19:24:30.927260 7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:45903 every 8 connection(s)
I20260810 19:24:30.928510 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:24:30.931937 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7533
I20260810 19:24:30.932099 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:24:30.932636 7340 external_mini_cluster.cc:1428] Setting key 31d31e61fbaffeabdf590f24949ac950
Aug 10 19:24:30 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389870, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.2@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:30.948978 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:30.931030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:56551 (local address 127.7.43.62:39265)
0810 19:24:30.931486 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:30.931491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:30.932436 (+ 945us) server_negotiation.cc:407] Connection header received
0810 19:24:30.934783 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:30.934787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:30.934852 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:30.934970 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:30.936596 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.937652 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.939192 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.939751 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.939874 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:30.943503 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:30.943509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:30.943512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:30.943544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:30.946034 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:30.946834 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:30.946837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:30.946840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:30.946962 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:30.947397 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:30.947400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:30.947402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:30.947646 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:30.947751 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:30.948531 (+ 780us) server_negotiation.cc:299] Negotiation successful
0810 19:24:30.948693 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260810 19:24:30.950199 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:30.931451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39265 (local address 127.7.43.2:56551)
0810 19:24:30.932261 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:30.932309 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:30.934500 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:30.935180 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:30.935211 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:30.935808 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:30.936417 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:30.936443 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.937787 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.937791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:30.939046 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:30.939059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:30.939936 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:30.939941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:30.940143 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:30.940770 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:30.940794 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:30.943249 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:30.946163 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:30.946172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:30.946189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:30.946740 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:30.947056 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:30.947059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:30.947062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:30.947297 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:30.947754 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:30.947762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:30.948166 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 19:24:30.948464 (+ 298us) client_negotiation.cc:240] Negotiation successful
0810 19:24:30.948983 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":320,"threads_started":1}
I20260810 19:24:30.951922 7751 heartbeater.cc:347] Connected to a master server at 127.7.43.62:39265
I20260810 19:24:30.952229 7751 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:30.952939 7751 heartbeater.cc:511] Master 127.7.43.62:39265 requested a full tablet report, sending...
I20260810 19:24:30.954479 7183 ts_manager.cc:194] Registered new tserver with Master: ababb62988d34cb9b0a12bf8ac90de28 (127.7.43.2:45903)
I20260810 19:24:30.955544 7183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.2@KRBTEST.COM'} at 127.7.43.2:56551
WARNING: no policy specified for kudu/127.7.43.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
Entry for principal kudu/127.7.43.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab.
I20260810 19:24:31.004416 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39265
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:31.344705 7763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:31.344998 7763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:31.345118 7763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:31.352670 7763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:31.352795 7763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:31.352968 7763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:24:31.362769 7763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.3.keytab
--principal=kudu/127.7.43.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:39265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:31.364063 7763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:31.365406 7763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:31.378252 7778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:31.378283 7776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:31.378532 7780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:31.379163 7763 server_base.cc:1034] running on GCE node
I20260810 19:24:31.379776 7763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:31.380728 7763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:31.381933 7763 hybrid_clock.cc:648] HybridClock initialized: now 1786389871381874 us; error 43 us; skew 500 ppm
I20260810 19:24:31.382241 7763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:31 dist-test-slave-7lbf krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389871, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:31.385762 7763 init.cc:377] Logged in from keytab as kudu/127.7.43.3@KRBTEST.COM (short username kudu)
I20260810 19:24:31.387696 7763 webserver.cc:460] Webserver started at http://127.7.43.3:44203/ using document root <none> and password file <none>
I20260810 19:24:31.388389 7763 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:31.388494 7763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:31.388767 7763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:31.391773 7763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "69eead16f28d44609d2fa0c11f46f2ea"
format_stamp: "Formatted at 2026-08-10 19:24:31 on dist-test-slave-7lbf"
server_key: "5927f6276bf4ef8f6808077077bb9db0"
server_key_iv: "2fc2477069ed7172a24fba3a32013e90"
server_key_version: "encryptionkey@0"
I20260810 19:24:31.392572 7763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "69eead16f28d44609d2fa0c11f46f2ea"
format_stamp: "Formatted at 2026-08-10 19:24:31 on dist-test-slave-7lbf"
server_key: "5927f6276bf4ef8f6808077077bb9db0"
server_key_iv: "2fc2477069ed7172a24fba3a32013e90"
server_key_version: "encryptionkey@0"
I20260810 19:24:31.398254 7763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:24:31.401809 7789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:31.403200 7763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:24:31.403354 7763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "69eead16f28d44609d2fa0c11f46f2ea"
format_stamp: "Formatted at 2026-08-10 19:24:31 on dist-test-slave-7lbf"
server_key: "5927f6276bf4ef8f6808077077bb9db0"
server_key_iv: "2fc2477069ed7172a24fba3a32013e90"
server_key_version: "encryptionkey@0"
I20260810 19:24:31.403555 7763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:31.419524 7763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:31.423681 7763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:31.423964 7763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:31.424943 7763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:31.426440 7763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:31.426514 7763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:31.426632 7763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:31.426679 7763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:31.428745 7526 heartbeater.cc:503] Master 127.7.43.62:39265 was elected leader, sending a full tablet report...
I20260810 19:24:31.457782 7763 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:35991
I20260810 19:24:31.457820 7975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:35991 every 8 connection(s)
I20260810 19:24:31.459071 7763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:24:31.467180 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 7763
I20260810 19:24:31.467342 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:24:31.467793 7340 external_mini_cluster.cc:1428] Setting key 730ddc0d41dec5a542222d5a5d91b79a
Aug 10 19:24:31 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389871, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.3@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:24:31.478900 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.461642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:34543 (local address 127.7.43.62:39265)
0810 19:24:31.462088 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.462092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.462983 (+ 891us) server_negotiation.cc:407] Connection header received
0810 19:24:31.464862 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.464869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.464973 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.465074 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:31.466677 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.467512 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.469239 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.469714 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.469826 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.473406 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:31.473412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:31.473415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:31.473448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:31.475910 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.476603 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.476608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.476612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.476741 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.477162 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.477167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.477170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.477452 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:31.477616 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.478285 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.478528 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260810 19:24:31.479861 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.462087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39265 (local address 127.7.43.3:34543)
0810 19:24:31.462807 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:31.462863 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:31.464581 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:31.465332 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:31.465341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:31.465916 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:31.466501 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.466520 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.467671 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.467675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.469104 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.469117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.469950 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.469958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.470181 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.470911 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:24:31.470935 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:31.473178 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:31.476041 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:31.476050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:31.476067 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:31.476491 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:31.476851 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:24:31.476856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:24:31.476859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:24:31.477054 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:24:31.477586 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:31.477593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:31.477932 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 19:24:31.478177 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 19:24:31.478692 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":273,"threads_started":1}
I20260810 19:24:31.481312 7976 heartbeater.cc:347] Connected to a master server at 127.7.43.62:39265
I20260810 19:24:31.481689 7976 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:31.482538 7976 heartbeater.cc:511] Master 127.7.43.62:39265 requested a full tablet report, sending...
I20260810 19:24:31.483932 7183 ts_manager.cc:194] Registered new tserver with Master: 69eead16f28d44609d2fa0c11f46f2ea (127.7.43.3:35991)
I20260810 19:24:31.484944 7183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.43.3@KRBTEST.COM'} at 127.7.43.3:34543
I20260810 19:24:31.495008 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:24:31.511448 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.499128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.7.43.62:39265)
0810 19:24:31.499323 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.499329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.499706 (+ 377us) server_negotiation.cc:407] Connection header received
0810 19:24:31.499903 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.499908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.500005 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.500091 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:31.501575 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.502517 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.503778 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.504199 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.504309 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.506727 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:31.506732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:31.506735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:31.506768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:31.509138 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.509794 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.509798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.509801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.509950 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.510244 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.510246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.510248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.510555 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:31.510735 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.511002 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.511174 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:24:31.515133 7183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:24:31.522137 7183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:24:31.543675 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.534242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45903 (local address 127.0.0.1:46352)
0810 19:24:31.534937 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:31.534959 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:31.535059 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:31.536513 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:31.536518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:31.536561 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:31.537002 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.537016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.538951 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.538956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.541182 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.541192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.542579 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.542585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.542774 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.542816 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 19:24:31.542950 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 19:24:31.543139 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":25,"thread_start_us":910,"threads_started":1}
I20260810 19:24:31.543843 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.534774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.7.43.2:45903)
0810 19:24:31.535806 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.535814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.535840 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:24:31.535930 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.535937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.536174 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.537275 (+ 1101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:31.537548 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.538826 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.541335 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.542387 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.542508 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.542717 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.543328 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.543517 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":719,"threads_started":1}
I20260810 19:24:31.543985 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.535258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:35991 (local address 127.0.0.1:59386)
0810 19:24:31.536092 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:31.536119 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:31.536244 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:31.537438 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:31.537443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:31.537466 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:31.537841 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.537851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.539604 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.539612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.541412 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.541427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.543148 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.543155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.543357 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.543391 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 19:24:31.543469 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 19:24:31.543591 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":40,"thread_start_us":945,"threads_started":1}
I20260810 19:24:31.544757 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.535843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.7.43.3:35991)
0810 19:24:31.536850 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.536859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.536890 (+ 31us) server_negotiation.cc:407] Connection header received
0810 19:24:31.536985 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.536992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.537208 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.537440 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:31.538004 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.539423 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.541589 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.542919 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.544065 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.544297 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.544372 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.544523 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":1294,"threads_started":1}
I20260810 19:24:31.545805 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.533810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.7.43.1:40753)
0810 19:24:31.534521 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.534529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.534941 (+ 412us) server_negotiation.cc:407] Connection header received
0810 19:24:31.535053 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.535062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.535324 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.535601 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:31.536627 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.538230 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.541364 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.542823 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.542947 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.543173 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.545326 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.545491 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":355,"threads_started":1}
I20260810 19:24:31.547464 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.533654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40753 (local address 127.0.0.1:49422)
0810 19:24:31.534472 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:31.534521 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:31.534713 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:31.535919 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:31.535926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:31.535961 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:31.536399 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.536415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.538363 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.538369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.541141 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.541169 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.544183 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.544189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.545156 (+ 967us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.545180 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:24:31.545262 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:24:31.545378 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":95,"thread_start_us":724,"threads_started":1}
I20260810 19:24:31.547758 7641 tablet_service.cc:1467] Processing CreateTablet for tablet 72d1191ca3c44426859c725a10c9a11e (DEFAULT_TABLE table=test-table [id=c42786f8438c4951a550e9b654b657df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:31.548079 7873 tablet_service.cc:1467] Processing CreateTablet for tablet 72d1191ca3c44426859c725a10c9a11e (DEFAULT_TABLE table=test-table [id=c42786f8438c4951a550e9b654b657df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:31.549299 7873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d1191ca3c44426859c725a10c9a11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:31.550294 7641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d1191ca3c44426859c725a10c9a11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:31.551087 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 72d1191ca3c44426859c725a10c9a11e (DEFAULT_TABLE table=test-table [id=c42786f8438c4951a550e9b654b657df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:31.552489 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d1191ca3c44426859c725a10c9a11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:31.560711 8001 tablet_bootstrap.cc:492] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Bootstrap starting.
I20260810 19:24:31.563787 8001 tablet_bootstrap.cc:654] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:31.564618 8003 tablet_bootstrap.cc:492] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Bootstrap starting.
I20260810 19:24:31.565107 8001 log.cc:824] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:31.567118 8007 tablet_bootstrap.cc:492] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Bootstrap starting.
I20260810 19:24:31.568574 8003 tablet_bootstrap.cc:654] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:31.568725 8001 tablet_bootstrap.cc:492] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: No bootstrap required, opened a new log
I20260810 19:24:31.568984 8001 ts_tablet_manager.cc:1397] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 19:24:31.569861 8003 log.cc:824] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:31.570695 8007 tablet_bootstrap.cc:654] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:31.571826 8007 log.cc:824] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:31.573123 8003 tablet_bootstrap.cc:492] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: No bootstrap required, opened a new log
I20260810 19:24:31.573443 8003 ts_tablet_manager.cc:1397] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 19:24:31.575868 8007 tablet_bootstrap.cc:492] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: No bootstrap required, opened a new log
I20260810 19:24:31.576138 8007 ts_tablet_manager.cc:1397] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 19:24:31.577284 8001 raft_consensus.cc:348] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.577680 8001 raft_consensus.cc:374] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:31.577788 8001 raft_consensus.cc:729] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69eead16f28d44609d2fa0c11f46f2ea, State: Initialized, Role: FOLLOWER
I20260810 19:24:31.578366 8001 consensus_queue.cc:260] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.579381 7976 heartbeater.cc:503] Master 127.7.43.62:39265 was elected leader, sending a full tablet report...
I20260810 19:24:31.579950 8001 ts_tablet_manager.cc:1428] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 19:24:31.584174 8003 raft_consensus.cc:348] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.584607 8003 raft_consensus.cc:374] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:31.584753 8003 raft_consensus.cc:729] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ababb62988d34cb9b0a12bf8ac90de28, State: Initialized, Role: FOLLOWER
I20260810 19:24:31.584846 8007 raft_consensus.cc:348] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.585309 8007 raft_consensus.cc:374] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:31.585438 8007 raft_consensus.cc:729] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ce1a120c44412abb39de9fec8c4b1a, State: Initialized, Role: FOLLOWER
I20260810 19:24:31.585672 8003 consensus_queue.cc:260] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.586221 8007 consensus_queue.cc:260] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.586766 7751 heartbeater.cc:503] Master 127.7.43.62:39265 was elected leader, sending a full tablet report...
I20260810 19:24:31.587486 8003 ts_tablet_manager.cc:1428] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260810 19:24:31.588060 8007 ts_tablet_manager.cc:1428] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260810 19:24:31.595597 8012 raft_consensus.cc:482] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:31.595854 8012 raft_consensus.cc:504] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.597467 8012 leader_election.cc:288] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85ce1a120c44412abb39de9fec8c4b1a (127.7.43.1:40753), ababb62988d34cb9b0a12bf8ac90de28 (127.7.43.2:45903)
I20260810 19:24:31.604019 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.597899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:55403 (local address 127.7.43.2:45903)
0810 19:24:31.598105 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.598111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.598181 (+ 70us) server_negotiation.cc:407] Connection header received
0810 19:24:31.598378 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.598384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.598460 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.598550 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:31.599123 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.600132 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.602071 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.603240 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.603335 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.603416 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.603729 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.603823 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:24:31.604115 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.597750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:45903 (local address 127.7.43.3:55403)
0810 19:24:31.598100 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:31.598119 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:31.598248 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:31.598670 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:31.598674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:31.598704 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:31.598963 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.598979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.600253 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.600258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.601910 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.601923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.603378 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.603383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.603582 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.603618 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 19:24:31.603709 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 19:24:31.603845 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260810 19:24:31.605165 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.597910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:52555 (local address 127.7.43.1:40753)
0810 19:24:31.598126 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.598132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.599297 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 19:24:31.599440 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.599444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.599500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.599567 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:31.600093 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.601082 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.603208 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.604372 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.604491 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.604586 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.604900 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.604997 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:24:31.605285 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.598439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:40753 (local address 127.7.43.3:52555)
0810 19:24:31.599189 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:31.599213 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:31.599336 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:31.599598 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:31.599603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:31.599626 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:31.599960 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.599971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.601237 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.601241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.603043 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:31.603060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.604571 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.604577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:31.604731 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.604761 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 19:24:31.604861 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:24:31.604981 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":375,"threads_started":1}
I20260810 19:24:31.605195 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d1191ca3c44426859c725a10c9a11e" candidate_uuid: "69eead16f28d44609d2fa0c11f46f2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ababb62988d34cb9b0a12bf8ac90de28" is_pre_election: true
I20260810 19:24:31.605746 7669 raft_consensus.cc:2454] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69eead16f28d44609d2fa0c11f46f2ea in term 0.
I20260810 19:24:31.606375 7796 leader_election.cc:302] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69eead16f28d44609d2fa0c11f46f2ea, ababb62988d34cb9b0a12bf8ac90de28; no voters:
I20260810 19:24:31.606328 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d1191ca3c44426859c725a10c9a11e" candidate_uuid: "69eead16f28d44609d2fa0c11f46f2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ce1a120c44412abb39de9fec8c4b1a" is_pre_election: true
I20260810 19:24:31.606752 8012 raft_consensus.cc:2781] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:24:31.606869 8012 raft_consensus.cc:482] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:24:31.606837 7439 raft_consensus.cc:2454] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69eead16f28d44609d2fa0c11f46f2ea in term 0.
I20260810 19:24:31.606962 8012 raft_consensus.cc:3037] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:31.609301 8012 raft_consensus.cc:504] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.609870 8012 leader_election.cc:288] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [CANDIDATE]: Term 1 election: Requested vote from peers 85ce1a120c44412abb39de9fec8c4b1a (127.7.43.1:40753), ababb62988d34cb9b0a12bf8ac90de28 (127.7.43.2:45903)
I20260810 19:24:31.610276 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d1191ca3c44426859c725a10c9a11e" candidate_uuid: "69eead16f28d44609d2fa0c11f46f2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ce1a120c44412abb39de9fec8c4b1a"
I20260810 19:24:31.610324 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d1191ca3c44426859c725a10c9a11e" candidate_uuid: "69eead16f28d44609d2fa0c11f46f2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ababb62988d34cb9b0a12bf8ac90de28"
I20260810 19:24:31.610450 7439 raft_consensus.cc:3037] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:31.610526 7669 raft_consensus.cc:3037] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:31.612632 7439 raft_consensus.cc:2454] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69eead16f28d44609d2fa0c11f46f2ea in term 1.
I20260810 19:24:31.612769 7669 raft_consensus.cc:2454] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69eead16f28d44609d2fa0c11f46f2ea in term 1.
I20260810 19:24:31.613013 7793 leader_election.cc:302] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69eead16f28d44609d2fa0c11f46f2ea, 85ce1a120c44412abb39de9fec8c4b1a; no voters:
I20260810 19:24:31.613338 8012 raft_consensus.cc:2781] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:31.613796 8012 raft_consensus.cc:686] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 1 LEADER]: Becoming Leader. State: Replica: 69eead16f28d44609d2fa0c11f46f2ea, State: Running, Role: LEADER
I20260810 19:24:31.614341 8012 consensus_queue.cc:237] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } }
I20260810 19:24:31.618717 7183 catalog_manager.cc:5495] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 69eead16f28d44609d2fa0c11f46f2ea (127.7.43.3). New cstate: current_term: 1 leader_uuid: "69eead16f28d44609d2fa0c11f46f2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69eead16f28d44609d2fa0c11f46f2ea" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 } health_report { overall_health: UNKNOWN } } }
I20260810 19:24:31.645920 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.640894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.7.43.3:35991)
0810 19:24:31.641106 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.641111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.641128 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:24:31.641276 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.641280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.641362 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.641492 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:24:31.641836 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.642572 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.643959 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.644346 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.644439 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.644899 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:24:31.645404 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:24:31.645546 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.645592 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.645659 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 19:24:31.646968 7527 tablet.cc:2369] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:24:31.654278 7669 raft_consensus.cc:1261] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 1 FOLLOWER]: Refusing update from remote peer 69eead16f28d44609d2fa0c11f46f2ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:31.654321 7439 raft_consensus.cc:1261] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 1 FOLLOWER]: Refusing update from remote peer 69eead16f28d44609d2fa0c11f46f2ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:31.655254 8019 consensus_queue.cc:1035] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ababb62988d34cb9b0a12bf8ac90de28" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:31.655608 8012 consensus_queue.cc:1035] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ce1a120c44412abb39de9fec8c4b1a" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:31.687161 7183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:31.687398 7183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:31.691171 7183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78d56ac65420424e971b63968b0b1416: Sending DeleteTablet for 3 replicas of tablet 72d1191ca3c44426859c725a10c9a11e
I20260810 19:24:31.692708 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d1191ca3c44426859c725a10c9a11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:31 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:49422
I20260810 19:24:31.692783 7873 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d1191ca3c44426859c725a10c9a11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:31 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:59386
I20260810 19:24:31.692739 7641 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d1191ca3c44426859c725a10c9a11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:31 UTC) from {username='kudu', principal='kudu/127.7.43.62@KRBTEST.COM'} at 127.0.0.1:46352
I20260810 19:24:31.694778 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7284
I20260810 19:24:31.709615 8042 tablet_replica.cc:330] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: stopping tablet replica
I20260810 19:24:31.713820 8042 raft_consensus.cc:2229] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:31.714622 8042 raft_consensus.cc:2258] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:31.718791 8042 ts_tablet_manager.cc:1905] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 19:24:31 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.1@KRBTEST.COM
I20260810 19:24:31.726152 8043 tablet_replica.cc:330] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: stopping tablet replica
I20260810 19:24:31.726342 8044 tablet_replica.cc:330] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: stopping tablet replica
I20260810 19:24:31.726804 8043 raft_consensus.cc:2229] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:24:31.726795 8044 raft_consensus.cc:2229] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:31.727357 8044 raft_consensus.cc:2258] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:31.727651 8043 raft_consensus.cc:2258] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:31.730515 8043 ts_tablet_manager.cc:1905] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:31.732836 8042 ts_tablet_manager.cc:1918] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:31.733054 8042 log.cc:1196] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-0/wal/wals/72d1191ca3c44426859c725a10c9a11e
I20260810 19:24:31.733652 8042 ts_tablet_manager.cc:1939] T 72d1191ca3c44426859c725a10c9a11e P 85ce1a120c44412abb39de9fec8c4b1a: Deleting consensus metadata
I20260810 19:24:31.735874 7157 catalog_manager.cc:4841] TS 85ce1a120c44412abb39de9fec8c4b1a (127.7.43.1:40753): tablet 72d1191ca3c44426859c725a10c9a11e (table test-table [id=c42786f8438c4951a550e9b654b657df]) successfully deleted
I20260810 19:24:31.735992 8044 ts_tablet_manager.cc:1905] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:31.737244 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.696324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.7.43.1:40753)
0810 19:24:31.696512 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.696519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.704187 (+ 7668us) server_negotiation.cc:407] Connection header received
0810 19:24:31.705574 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.705581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.705698 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.705786 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:31.709959 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.711195 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.715157 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.715905 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.716006 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.727323 (+ 11317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:31.727340 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:31.727358 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:31.727412 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:31.731498 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.733405 (+ 1907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.733415 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.733419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.733633 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.734463 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.734469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.734472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.734997 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:31.735289 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.736621 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.736862 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:24:31.738179 8043 ts_tablet_manager.cc:1918] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:31.738369 8043 log.cc:1196] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/72d1191ca3c44426859c725a10c9a11e
I20260810 19:24:31.738813 7511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:31.739984 8043 ts_tablet_manager.cc:1939] T 72d1191ca3c44426859c725a10c9a11e P 69eead16f28d44609d2fa0c11f46f2ea: Deleting consensus metadata
I20260810 19:24:31.741927 7157 catalog_manager.cc:4841] TS 69eead16f28d44609d2fa0c11f46f2ea (127.7.43.3:35991): tablet 72d1191ca3c44426859c725a10c9a11e (table test-table [id=c42786f8438c4951a550e9b654b657df]) successfully deleted
I20260810 19:24:31.742918 8044 ts_tablet_manager.cc:1918] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:31.743141 8044 log.cc:1196] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/72d1191ca3c44426859c725a10c9a11e
I20260810 19:24:31.743814 8044 ts_tablet_manager.cc:1939] T 72d1191ca3c44426859c725a10c9a11e P ababb62988d34cb9b0a12bf8ac90de28: Deleting consensus metadata
I20260810 19:24:31.745939 7162 catalog_manager.cc:4841] TS ababb62988d34cb9b0a12bf8ac90de28 (127.7.43.2:45903): tablet 72d1191ca3c44426859c725a10c9a11e (table test-table [id=c42786f8438c4951a550e9b654b657df]) successfully deleted
I20260810 19:24:31.843555 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7284
I20260810 19:24:31.877223 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7533
Aug 10 19:24:31 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.2@KRBTEST.COM
I20260810 19:24:31.894580 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:31.878085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.7.43.2:45903)
0810 19:24:31.878237 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:31.878242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:31.878261 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:24:31.878434 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:31.878438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:31.878504 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:31.878592 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:31.880190 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.880921 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.882972 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:31.883380 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:31.883475 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:31.888575 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:31.888591 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:31.888607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:31.888647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:31.891530 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.892396 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.892407 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.892409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.892566 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:31.892982 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:31.892985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:31.892993 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:31.893388 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:31.893656 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:31.894096 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 19:24:31.894309 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 19:24:31.895329 7745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:31.981952 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7533
I20260810 19:24:32.008451 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7763
Aug 10 19:24:32 dist-test-slave-7lbf krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.3@KRBTEST.COM
I20260810 19:24:32.027578 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:32.009328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.7.43.3:35991)
0810 19:24:32.009501 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:32.009506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:32.009628 (+ 122us) server_negotiation.cc:407] Connection header received
0810 19:24:32.009832 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:32.009836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:32.009906 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:32.010002 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:32.011939 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:32.012780 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:32.014988 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:32.015426 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:32.015546 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:32.021136 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:32.021150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:32.021172 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:32.021213 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:32.024410 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:32.025280 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:32.025289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:32.025291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:32.025464 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:32.025973 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:32.025976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:32.025978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:32.026406 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:32.026655 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:32.027048 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 19:24:32.027224 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:24:32.028463 7966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:32.151508 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7763
I20260810 19:24:32.175822 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 7124
I20260810 19:24:32.176949 7231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:32.286029 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 7124
2026-08-10T19:24:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4539 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T19:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:32Z Disabled control of system clock
I20260810 19:24:32.385921 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43971
--webserver_interface=127.7.43.62
--webserver_port=0
--builtin_ntp_servers=127.7.43.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:43971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:24:32.736421 8060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:32.736733 8060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:32.736841 8060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:32.744323 8060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:32.744458 8060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:32.744516 8060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:32.744582 8060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:32.744634 8060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:32.754333 8060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:43971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:43971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:32.755683 8060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:32.757747 8060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:32.770720 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:32.770803 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:32.770774 8072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:32.771721 8060 server_base.cc:1034] running on GCE node
I20260810 19:24:32.772482 8060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:32.774057 8060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:32.775257 8060 hybrid_clock.cc:648] HybridClock initialized: now 1786389872775208 us; error 36 us; skew 500 ppm
I20260810 19:24:32.775631 8060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:32.781142 8060 webserver.cc:460] Webserver started at http://127.7.43.62:34917/ using document root <none> and password file <none>
I20260810 19:24:32.781948 8060 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:32.782045 8060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:32.782373 8060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:32.785498 8060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "4dd1baa1efb348168882885d07ee3c83"
format_stamp: "Formatted at 2026-08-10 19:24:32 on dist-test-slave-7lbf"
server_key: "3a43d230ade0bf6850e7504b844b035d"
server_key_iv: "6de07aafb8d8c05e0915346b6251a765"
server_key_version: "encryptionkey@0"
I20260810 19:24:32.786389 8060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "4dd1baa1efb348168882885d07ee3c83"
format_stamp: "Formatted at 2026-08-10 19:24:32 on dist-test-slave-7lbf"
server_key: "3a43d230ade0bf6850e7504b844b035d"
server_key_iv: "6de07aafb8d8c05e0915346b6251a765"
server_key_version: "encryptionkey@0"
I20260810 19:24:32.792394 8060 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260810 19:24:32.796077 8081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:32.797406 8060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:24:32.797632 8060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "4dd1baa1efb348168882885d07ee3c83"
format_stamp: "Formatted at 2026-08-10 19:24:32 on dist-test-slave-7lbf"
server_key: "3a43d230ade0bf6850e7504b844b035d"
server_key_iv: "6de07aafb8d8c05e0915346b6251a765"
server_key_version: "encryptionkey@0"
I20260810 19:24:32.797835 8060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:32.813402 8060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:32.814285 8060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:32.814569 8060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:32.832631 8060 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:43971
I20260810 19:24:32.832638 8170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:43971 every 8 connection(s)
I20260810 19:24:32.834025 8060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:32.837879 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:32.838349 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8060
I20260810 19:24:32.838497 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:32.838951 7340 external_mini_cluster.cc:1428] Setting key 1069f81a87ca95427acd7a61ae612977
I20260810 19:24:32.850320 8171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83: Bootstrap starting.
I20260810 19:24:32.852530 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:32.841039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.7.43.62:43971)
0810 19:24:32.842180 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:32.842191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:32.842235 (+ 44us) server_negotiation.cc:407] Connection header received
0810 19:24:32.844093 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:32.844117 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:32.844596 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:32.844984 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:32.845584 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:32.846828 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:32.848166 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:32.848868 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:32.849068 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:32.850203 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:32.850210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:32.850225 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:32.850802 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:32.850896 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:32.850923 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:32.851034 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:32.851255 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 19:24:32.851558 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":375,"threads_started":1}
I20260810 19:24:32.853948 8171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:32.855039 8171 log.cc:824] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:32.857759 8171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83: No bootstrap required, opened a new log
I20260810 19:24:32.865199 8171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd1baa1efb348168882885d07ee3c83" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43971 } }
I20260810 19:24:32.865602 8171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:32.865696 8171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd1baa1efb348168882885d07ee3c83, State: Initialized, Role: FOLLOWER
I20260810 19:24:32.866240 8171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd1baa1efb348168882885d07ee3c83" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43971 } }
I20260810 19:24:32.866487 8171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:32.866611 8171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:32.866770 8171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:32.869297 8171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd1baa1efb348168882885d07ee3c83" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43971 } }
I20260810 19:24:32.869853 8171 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd1baa1efb348168882885d07ee3c83; no voters:
I20260810 19:24:32.870422 8171 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:32.870538 8177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:32.871241 8177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [term 1 LEADER]: Becoming Leader. State: Replica: 4dd1baa1efb348168882885d07ee3c83, State: Running, Role: LEADER
I20260810 19:24:32.871768 8177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd1baa1efb348168882885d07ee3c83" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43971 } }
I20260810 19:24:32.872435 8171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:32.874725 8178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd1baa1efb348168882885d07ee3c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd1baa1efb348168882885d07ee3c83" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43971 } } }
I20260810 19:24:32.874691 8179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd1baa1efb348168882885d07ee3c83. Latest consensus state: current_term: 1 leader_uuid: "4dd1baa1efb348168882885d07ee3c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd1baa1efb348168882885d07ee3c83" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 43971 } } }
I20260810 19:24:32.875222 8178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:32.875293 8179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:32.878840 8188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:32.885087 8188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:32.894912 8188 catalog_manager.cc:1347] Generated new cluster ID: b5568176906840448272b7525a74bbb9
I20260810 19:24:32.895040 8188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:32.907852 8188 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:32.909667 8188 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:32.917338 8188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83: Generated new TSK 0
I20260810 19:24:32.918447 8188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:32.925705 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.1:0
--local_ip_for_outbound_sockets=127.7.43.1
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43971
--builtin_ntp_servers=127.7.43.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:24:33.268220 8205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:33.268510 8205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:33.268622 8205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:33.275936 8205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:33.276043 8205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:33.276199 8205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.1
I20260810 19:24:33.285423 8205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.43.1
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.1
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:33.286738 8205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:33.288506 8205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:33.301352 8213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:33.301407 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:33.301473 8220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:33.302060 8205 server_base.cc:1034] running on GCE node
I20260810 19:24:33.302692 8205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:33.303601 8205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:33.304798 8205 hybrid_clock.cc:648] HybridClock initialized: now 1786389873304743 us; error 39 us; skew 500 ppm
I20260810 19:24:33.305151 8205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:33.307418 8205 webserver.cc:460] Webserver started at http://127.7.43.1:45255/ using document root <none> and password file <none>
I20260810 19:24:33.308120 8205 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:33.308214 8205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:33.308552 8205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:33.311563 8205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data/instance:
uuid: "7474eb9830654b819e8b7903ede8d3b0"
format_stamp: "Formatted at 2026-08-10 19:24:33 on dist-test-slave-7lbf"
server_key: "82e0c5583c5c3c304a57041974beb59c"
server_key_iv: "15bcdd2e2a4d465976609cb08d6128dc"
server_key_version: "encryptionkey@0"
I20260810 19:24:33.312332 8205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance:
uuid: "7474eb9830654b819e8b7903ede8d3b0"
format_stamp: "Formatted at 2026-08-10 19:24:33 on dist-test-slave-7lbf"
server_key: "82e0c5583c5c3c304a57041974beb59c"
server_key_iv: "15bcdd2e2a4d465976609cb08d6128dc"
server_key_version: "encryptionkey@0"
I20260810 19:24:33.317355 8205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:24:33.320780 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:33.322157 8205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:33.322304 8205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal
uuid: "7474eb9830654b819e8b7903ede8d3b0"
format_stamp: "Formatted at 2026-08-10 19:24:33 on dist-test-slave-7lbf"
server_key: "82e0c5583c5c3c304a57041974beb59c"
server_key_iv: "15bcdd2e2a4d465976609cb08d6128dc"
server_key_version: "encryptionkey@0"
I20260810 19:24:33.322510 8205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:33.338395 8205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:33.339229 8205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:33.339501 8205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:33.340457 8205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:33.341956 8205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:33.342063 8205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:33.342170 8205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:33.342242 8205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:33.372922 8205 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.1:42561
I20260810 19:24:33.372990 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.1:42561 every 8 connection(s)
I20260810 19:24:33.374444 8205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/data/info.pb
I20260810 19:24:33.379902 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8205
I20260810 19:24:33.380060 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-0/wal/instance
I20260810 19:24:33.380476 7340 external_mini_cluster.cc:1428] Setting key a8caef721676161a607d2e335e949fb6
I20260810 19:24:33.383116 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.2:0
--local_ip_for_outbound_sockets=127.7.43.2
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43971
--builtin_ntp_servers=127.7.43.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:24:33.387065 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:33.377073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.1:54839 (local address 127.7.43.62:43971)
0810 19:24:33.377610 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:33.377615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:33.378726 (+ 1111us) server_negotiation.cc:407] Connection header received
0810 19:24:33.380678 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:33.380684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:33.380757 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:33.380856 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:33.382036 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.382773 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.384364 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.384750 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.384860 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:33.386257 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:33.386262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:33.386265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:33.386282 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:33.386325 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:33.386344 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:33.386417 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:33.386640 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 19:24:33.386735 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":302,"threads_started":1}
I20260810 19:24:33.388041 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:33.377642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43971 (local address 127.7.43.1:54839)
0810 19:24:33.378569 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:33.378613 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:33.380383 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:33.381114 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:33.381129 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:33.381219 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:33.381877 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:33.381896 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.382928 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.382932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:33.384219 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:33.384231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.384970 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.384978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:33.385200 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:33.385921 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:33.385925 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:33.385979 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:33.386075 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:33.386429 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:33.386442 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:33.386450 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:24:33.386599 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 19:24:33.387010 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":279,"threads_started":1}
I20260810 19:24:33.389539 8409 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43971
I20260810 19:24:33.389819 8409 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:33.390583 8409 heartbeater.cc:511] Master 127.7.43.62:43971 requested a full tablet report, sending...
I20260810 19:24:33.393240 8111 ts_manager.cc:194] Registered new tserver with Master: 7474eb9830654b819e8b7903ede8d3b0 (127.7.43.1:42561)
I20260810 19:24:33.395107 8111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.1:54839
W20260810 19:24:33.722337 8417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:33.722635 8417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:33.722723 8417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:33.730078 8417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:33.730183 8417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:33.730289 8417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.2
I20260810 19:24:33.739727 8417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.43.2
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.2
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:33.741012 8417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:33.742779 8417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:33.755667 8428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:33.755883 8423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:33.755883 8424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:33.756901 8417 server_base.cc:1034] running on GCE node
I20260810 19:24:33.757560 8417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:33.758560 8417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:33.759773 8417 hybrid_clock.cc:648] HybridClock initialized: now 1786389873759708 us; error 61 us; skew 500 ppm
I20260810 19:24:33.760105 8417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:33.762521 8417 webserver.cc:460] Webserver started at http://127.7.43.2:43149/ using document root <none> and password file <none>
I20260810 19:24:33.763262 8417 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:33.763370 8417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:33.763690 8417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:33.766752 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data/instance:
uuid: "cb1c70b29fa44bb4ae71695e534b63ca"
format_stamp: "Formatted at 2026-08-10 19:24:33 on dist-test-slave-7lbf"
server_key: "c890540a3e2d25c4ecc2f0c56c28974e"
server_key_iv: "168de91a041d2379742f2465ca51db4d"
server_key_version: "encryptionkey@0"
I20260810 19:24:33.767539 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance:
uuid: "cb1c70b29fa44bb4ae71695e534b63ca"
format_stamp: "Formatted at 2026-08-10 19:24:33 on dist-test-slave-7lbf"
server_key: "c890540a3e2d25c4ecc2f0c56c28974e"
server_key_iv: "168de91a041d2379742f2465ca51db4d"
server_key_version: "encryptionkey@0"
I20260810 19:24:33.772805 8417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 19:24:33.776499 8436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:33.778009 8417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:33.778178 8417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal
uuid: "cb1c70b29fa44bb4ae71695e534b63ca"
format_stamp: "Formatted at 2026-08-10 19:24:33 on dist-test-slave-7lbf"
server_key: "c890540a3e2d25c4ecc2f0c56c28974e"
server_key_iv: "168de91a041d2379742f2465ca51db4d"
server_key_version: "encryptionkey@0"
I20260810 19:24:33.778391 8417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:33.799382 8417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:33.800272 8417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:33.800551 8417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:33.801640 8417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:33.803210 8417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:33.803313 8417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:33.803416 8417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:33.803488 8417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:33.836318 8417 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.2:36189
I20260810 19:24:33.836377 8624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.2:36189 every 8 connection(s)
I20260810 19:24:33.837738 8417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/data/info.pb
I20260810 19:24:33.847635 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8417
I20260810 19:24:33.847795 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal/instance
I20260810 19:24:33.848235 7340 external_mini_cluster.cc:1428] Setting key e2ba7e2014070feec6e8daef4602bd64
I20260810 19:24:33.850776 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:33.840510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:41393 (local address 127.7.43.62:43971)
0810 19:24:33.840713 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:33.840719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:33.842289 (+ 1570us) server_negotiation.cc:407] Connection header received
0810 19:24:33.844083 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:33.844090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:33.844171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:33.844279 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:33.845317 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.846478 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.848080 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.848501 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.848612 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:33.849995 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:33.850000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:33.850003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:33.850021 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:33.850061 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:33.850073 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:33.850147 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:33.850361 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 19:24:33.850461 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:24:33.851178 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.43.3:0
--local_ip_for_outbound_sockets=127.7.43.3
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43971
--builtin_ntp_servers=127.7.43.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:24:33.851871 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:33.841034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43971 (local address 127.7.43.2:41393)
0810 19:24:33.842112 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:33.842165 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:33.843810 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:33.844471 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:33.844478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:33.844582 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:33.845158 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:33.845175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.846598 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.846602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:33.847941 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:33.847954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:33.848730 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:33.848739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:33.848957 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:33.849646 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:33.849649 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:33.849703 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:33.849812 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:33.850158 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:33.850170 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:33.850179 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 19:24:33.850335 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 19:24:33.850767 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":336,"threads_started":1}
I20260810 19:24:33.853328 8628 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43971
I20260810 19:24:33.853655 8628 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:33.854411 8628 heartbeater.cc:511] Master 127.7.43.62:43971 requested a full tablet report, sending...
I20260810 19:24:33.855794 8111 ts_manager.cc:194] Registered new tserver with Master: cb1c70b29fa44bb4ae71695e534b63ca (127.7.43.2:36189)
I20260810 19:24:33.856583 8111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.2:41393
W20260810 19:24:34.190462 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:34.190761 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:34.190884 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:34.198086 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:34.198190 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:34.198329 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.43.3
I20260810 19:24:34.207532 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.43.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.43.3
--webserver_port=0
--tserver_master_addrs=127.7.43.62:43971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.43.3
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:34.208863 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:34.210726 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:34.223780 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:34.223815 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:34.223891 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:34.224485 8634 server_base.cc:1034] running on GCE node
I20260810 19:24:34.225183 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:34.226285 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:34.227495 8634 hybrid_clock.cc:648] HybridClock initialized: now 1786389874227441 us; error 60 us; skew 500 ppm
I20260810 19:24:34.227818 8634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 19:24:34.230279 8634 webserver.cc:460] Webserver started at http://127.7.43.3:40235/ using document root <none> and password file <none>
I20260810 19:24:34.231017 8634 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:34.231114 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:34.231451 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:34.234711 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data/instance:
uuid: "8925240c42d343e5a3460b256a5b0fc8"
format_stamp: "Formatted at 2026-08-10 19:24:34 on dist-test-slave-7lbf"
server_key: "8a24f723b6210221e400df74c840e8be"
server_key_iv: "805b9dacb8e598dc86540c5369c0bb1a"
server_key_version: "encryptionkey@0"
I20260810 19:24:34.235541 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance:
uuid: "8925240c42d343e5a3460b256a5b0fc8"
format_stamp: "Formatted at 2026-08-10 19:24:34 on dist-test-slave-7lbf"
server_key: "8a24f723b6210221e400df74c840e8be"
server_key_iv: "805b9dacb8e598dc86540c5369c0bb1a"
server_key_version: "encryptionkey@0"
I20260810 19:24:34.240921 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:24:34.244737 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:34.246157 8634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:34.246320 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal
uuid: "8925240c42d343e5a3460b256a5b0fc8"
format_stamp: "Formatted at 2026-08-10 19:24:34 on dist-test-slave-7lbf"
server_key: "8a24f723b6210221e400df74c840e8be"
server_key_iv: "805b9dacb8e598dc86540c5369c0bb1a"
server_key_version: "encryptionkey@0"
I20260810 19:24:34.246528 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:34.268893 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:34.269795 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:34.270066 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:34.271106 8634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:24:34.272706 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:24:34.272814 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:34.272944 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:24:34.273026 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:34.303680 8634 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.3:44519
I20260810 19:24:34.303712 8857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.3:44519 every 8 connection(s)
I20260810 19:24:34.304946 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/data/info.pb
I20260810 19:24:34.314040 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8634
I20260810 19:24:34.314206 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal/instance
I20260810 19:24:34.314620 7340 external_mini_cluster.cc:1428] Setting key a00edd099c0b280bce2af55ee26ac294
I20260810 19:24:34.317009 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.307457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.3:39591 (local address 127.7.43.62:43971)
0810 19:24:34.307617 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.307622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.309062 (+ 1440us) server_negotiation.cc:407] Connection header received
0810 19:24:34.310889 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.310895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.310979 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.311074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:34.312149 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.312901 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.314544 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.314976 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.315104 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.316293 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:34.316304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:34.316308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:34.316328 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:34.316368 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:34.316385 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:34.316466 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.316681 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.316758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:24:34.318204 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.308011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:43971 (local address 127.7.43.3:39591)
0810 19:24:34.308902 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.308948 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:34.310622 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:34.311319 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:34.311327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:34.311414 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:24:34.311984 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.312003 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.313052 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.313056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.314411 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.314424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.315177 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.315182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.315397 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.316024 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:24:34.316026 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:24:34.316064 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:24:34.316143 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:24:34.316483 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:24:34.316495 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:24:34.316503 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:24:34.316678 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 19:24:34.317113 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":265,"threads_started":1}
I20260810 19:24:34.319365 8858 heartbeater.cc:347] Connected to a master server at 127.7.43.62:43971
I20260810 19:24:34.319613 8858 heartbeater.cc:464] Registering TS with master...
I20260810 19:24:34.320281 8858 heartbeater.cc:511] Master 127.7.43.62:43971 requested a full tablet report, sending...
I20260810 19:24:34.321584 8111 ts_manager.cc:194] Registered new tserver with Master: 8925240c42d343e5a3460b256a5b0fc8 (127.7.43.3:44519)
I20260810 19:24:34.322393 8111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.43.3:39591
I20260810 19:24:34.329640 7340 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:24:34.339015 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.333594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.7.43.62:43971)
0810 19:24:34.333755 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.333760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.334193 (+ 433us) server_negotiation.cc:407] Connection header received
0810 19:24:34.334319 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.334323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.334402 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.334482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:34.335013 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.335708 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.337041 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.337428 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.337584 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.338478 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:34.338483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:34.338487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:34.338504 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:34.338538 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:34.338549 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:34.338616 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.338707 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.338779 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:24:34.342635 8111 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:24:34.349393 8111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:24:34.385380 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.361221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.2:36189 (local address 127.0.0.1:56892)
0810 19:24:34.374351 (+ 13130us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.374407 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:34.374599 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:34.376927 (+ 2328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:34.376936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:34.376967 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:34.377383 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.377400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.379470 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.379479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.382011 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.382027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.384424 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.384430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.384672 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.384723 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 19:24:34.384832 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 19:24:34.385022 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12809,"thread_start_us":15864,"threads_started":1}
I20260810 19:24:34.386739 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.361896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.7.43.2:36189)
0810 19:24:34.376369 (+ 14473us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.376377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.376405 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:24:34.376483 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.376489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.376685 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.376862 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:34.377654 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.379309 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.382249 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.383890 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.384021 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.384284 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.386204 (+ 1920us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.386384 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14342,"thread_start_us":14233,"threads_started":1}
I20260810 19:24:34.388090 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.361781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.7.43.3:44519)
0810 19:24:34.375957 (+ 14176us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.375968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.378284 (+ 2316us) server_negotiation.cc:407] Connection header received
0810 19:24:34.378434 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.378473 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.378709 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.378895 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:34.379947 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.381582 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.385271 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.386668 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.386796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.387077 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.387576 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.387750 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13970,"thread_start_us":25389,"threads_started":1}
I20260810 19:24:34.388109 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.361559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:44519 (local address 127.0.0.1:47242)
0810 19:24:34.377750 (+ 16191us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.377775 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:34.377897 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:34.379181 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:34.379186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:34.379211 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:34.379738 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.379755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.381731 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.381736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.385049 (+ 3313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.385064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.387213 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.387219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.387385 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.387410 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:24:34.387489 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:24:34.387609 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16053,"thread_start_us":13233,"threads_started":1}
I20260810 19:24:34.392645 8731 tablet_service.cc:1467] Processing CreateTablet for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb (DEFAULT_TABLE table=test-table [id=177c5176f73844d2a23faa515f5e5d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:34.393437 8517 tablet_service.cc:1467] Processing CreateTablet for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb (DEFAULT_TABLE table=test-table [id=177c5176f73844d2a23faa515f5e5d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:34.394157 8731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:34.394984 8517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:34.396118 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.377546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44738)
0810 19:24:34.385916 (+ 8370us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.385936 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:34.386033 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:34.389335 (+ 3302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:34.389340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:34.389365 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:34.389570 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.389583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.391485 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.391503 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.393232 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.393249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.395307 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.395314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.395514 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.395547 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:24:34.395640 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 19:24:34.395783 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8252,"mutex_wait_us":427,"thread_start_us":11688,"threads_started":1}
I20260810 19:24:34.396966 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.377817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.7.43.1:42561)
0810 19:24:34.388845 (+ 11028us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.388852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.388876 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:24:34.388952 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.388959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.389157 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.389324 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:34.389751 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.391296 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.393429 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.394810 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.394943 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.395202 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.396445 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.396615 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10887,"thread_start_us":11608,"threads_started":1}
I20260810 19:24:34.398520 8409 heartbeater.cc:503] Master 127.7.43.62:43971 was elected leader, sending a full tablet report...
I20260810 19:24:34.401072 8304 tablet_service.cc:1467] Processing CreateTablet for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb (DEFAULT_TABLE table=test-table [id=177c5176f73844d2a23faa515f5e5d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:24:34.402714 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:34.407563 8895 tablet_bootstrap.cc:492] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Bootstrap starting.
I20260810 19:24:34.409647 8896 tablet_bootstrap.cc:492] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Bootstrap starting.
I20260810 19:24:34.411567 8895 tablet_bootstrap.cc:654] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:34.413129 8895 log.cc:824] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:34.413484 8896 tablet_bootstrap.cc:654] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:34.414608 8896 log.cc:824] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:34.416443 8895 tablet_bootstrap.cc:492] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: No bootstrap required, opened a new log
I20260810 19:24:34.416707 8902 tablet_bootstrap.cc:492] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: Bootstrap starting.
I20260810 19:24:34.416857 8895 ts_tablet_manager.cc:1397] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260810 19:24:34.418633 8896 tablet_bootstrap.cc:492] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: No bootstrap required, opened a new log
I20260810 19:24:34.418923 8896 ts_tablet_manager.cc:1397] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 19:24:34.419723 8902 tablet_bootstrap.cc:654] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:34.420662 8902 log.cc:824] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:34.423182 8902 tablet_bootstrap.cc:492] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: No bootstrap required, opened a new log
I20260810 19:24:34.423420 8902 ts_tablet_manager.cc:1397] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 19:24:34.427577 8895 raft_consensus.cc:348] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.428284 8895 raft_consensus.cc:374] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:34.428412 8895 raft_consensus.cc:729] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8925240c42d343e5a3460b256a5b0fc8, State: Initialized, Role: FOLLOWER
I20260810 19:24:34.429107 8895 consensus_queue.cc:260] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.429270 8896 raft_consensus.cc:348] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.429819 8896 raft_consensus.cc:374] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:34.429981 8896 raft_consensus.cc:729] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1c70b29fa44bb4ae71695e534b63ca, State: Initialized, Role: FOLLOWER
I20260810 19:24:34.430332 8858 heartbeater.cc:503] Master 127.7.43.62:43971 was elected leader, sending a full tablet report...
I20260810 19:24:34.431177 8895 ts_tablet_manager.cc:1428] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260810 19:24:34.431524 8902 raft_consensus.cc:348] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.431982 8902 raft_consensus.cc:374] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:34.432103 8902 raft_consensus.cc:729] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7474eb9830654b819e8b7903ede8d3b0, State: Initialized, Role: FOLLOWER
I20260810 19:24:34.432227 8896 consensus_queue.cc:260] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.432701 8902 consensus_queue.cc:260] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.433390 8628 heartbeater.cc:503] Master 127.7.43.62:43971 was elected leader, sending a full tablet report...
I20260810 19:24:34.434085 8896 ts_tablet_manager.cc:1428] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260810 19:24:34.435792 8902 ts_tablet_manager.cc:1428] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260810 19:24:34.556596 8859 tablet.cc:2369] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:24:34.590065 8632 tablet.cc:2369] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:24:34.626825 8412 tablet.cc:2369] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:24:34.627772 8905 raft_consensus.cc:482] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:24:34.628103 8905 raft_consensus.cc:504] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.629994 8905 leader_election.cc:288] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8925240c42d343e5a3460b256a5b0fc8 (127.7.43.3:44519), 7474eb9830654b819e8b7903ede8d3b0 (127.7.43.1:42561)
I20260810 19:24:34.636770 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.630319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:54509 (local address 127.7.43.3:44519)
0810 19:24:34.630610 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.630619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.631438 (+ 819us) server_negotiation.cc:407] Connection header received
0810 19:24:34.631653 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.631657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.631719 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.631798 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:34.632459 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.633173 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.635066 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.635906 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.636031 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.636130 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.636427 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.636548 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 19:24:34.636863 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.630208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.3:44519 (local address 127.7.43.2:54509)
0810 19:24:34.631325 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.631343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:34.631497 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:34.631831 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:34.631836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:34.631870 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:34.632318 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.632334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.633303 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.633308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.634881 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.634894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.636102 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.636108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.636249 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.636277 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:24:34.636355 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 19:24:34.636477 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":20,"spinlock_wait_cycles":23424,"thread_start_us":825,"threads_started":1}
I20260810 19:24:34.637017 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.630535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.43.2:40217 (local address 127.7.43.1:42561)
0810 19:24:34.630770 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.630777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.631498 (+ 721us) server_negotiation.cc:407] Connection header received
0810 19:24:34.631671 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.631675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.631770 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.631877 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:24:34.632636 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.633337 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.635236 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.636080 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.636212 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.636330 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.636620 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.636747 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:24:34.637210 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.630421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.7.43.2:40217)
0810 19:24:34.631378 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.631393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:24:34.631535 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:24:34.631897 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:24:34.631902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:24:34.631953 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:24:34.632463 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.632475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.633489 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.633494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.635033 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:24:34.635046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.636281 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.636288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:24:34.636457 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.636482 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:24:34.636563 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:24:34.636685 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":1,"thread_start_us":489,"threads_started":1}
I20260810 19:24:34.638113 8772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eb3e8be72949fcb10c7ad4ac97e2eb" candidate_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8925240c42d343e5a3460b256a5b0fc8" is_pre_election: true
I20260810 19:24:34.638247 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eb3e8be72949fcb10c7ad4ac97e2eb" candidate_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7474eb9830654b819e8b7903ede8d3b0" is_pre_election: true
I20260810 19:24:34.638757 8331 raft_consensus.cc:2454] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb1c70b29fa44bb4ae71695e534b63ca in term 0.
I20260810 19:24:34.638753 8772 raft_consensus.cc:2454] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb1c70b29fa44bb4ae71695e534b63ca in term 0.
I20260810 19:24:34.639434 8440 leader_election.cc:302] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8925240c42d343e5a3460b256a5b0fc8, cb1c70b29fa44bb4ae71695e534b63ca; no voters:
I20260810 19:24:34.639815 8905 raft_consensus.cc:2781] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:24:34.639957 8905 raft_consensus.cc:482] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:24:34.640060 8905 raft_consensus.cc:3037] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:34.642387 8905 raft_consensus.cc:504] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.642978 8905 leader_election.cc:288] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [CANDIDATE]: Term 1 election: Requested vote from peers 8925240c42d343e5a3460b256a5b0fc8 (127.7.43.3:44519), 7474eb9830654b819e8b7903ede8d3b0 (127.7.43.1:42561)
I20260810 19:24:34.643424 8772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eb3e8be72949fcb10c7ad4ac97e2eb" candidate_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8925240c42d343e5a3460b256a5b0fc8"
I20260810 19:24:34.643437 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eb3e8be72949fcb10c7ad4ac97e2eb" candidate_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7474eb9830654b819e8b7903ede8d3b0"
I20260810 19:24:34.643612 8331 raft_consensus.cc:3037] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:34.643620 8772 raft_consensus.cc:3037] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:34.646003 8331 raft_consensus.cc:2454] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1c70b29fa44bb4ae71695e534b63ca in term 1.
I20260810 19:24:34.646061 8772 raft_consensus.cc:2454] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1c70b29fa44bb4ae71695e534b63ca in term 1.
I20260810 19:24:34.646420 8441 leader_election.cc:302] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7474eb9830654b819e8b7903ede8d3b0, cb1c70b29fa44bb4ae71695e534b63ca; no voters:
I20260810 19:24:34.646739 8905 raft_consensus.cc:2781] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:34.647184 8905 raft_consensus.cc:686] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 1 LEADER]: Becoming Leader. State: Replica: cb1c70b29fa44bb4ae71695e534b63ca, State: Running, Role: LEADER
I20260810 19:24:34.647707 8905 consensus_queue.cc:237] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } }
I20260810 19:24:34.652346 8108 catalog_manager.cc:5495] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca reported cstate change: term changed from 0 to 1, leader changed from <none> to cb1c70b29fa44bb4ae71695e534b63ca (127.7.43.2). New cstate: current_term: 1 leader_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1c70b29fa44bb4ae71695e534b63ca" member_type: VOTER last_known_addr { host: "127.7.43.2" port: 36189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260810 19:24:34.738971 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.733506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.7.43.2:36189)
0810 19:24:34.733709 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.733715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.733761 (+ 46us) server_negotiation.cc:407] Connection header received
0810 19:24:34.734011 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.734015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.734107 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.734241 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:24:34.734653 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.735393 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.736802 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.737240 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.737351 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.737961 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:24:34.738466 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:24:34.738597 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.738686 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.738757 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:24:34.747052 8331 raft_consensus.cc:1261] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 1 FOLLOWER]: Refusing update from remote peer cb1c70b29fa44bb4ae71695e534b63ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:34.747049 8772 raft_consensus.cc:1261] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 1 FOLLOWER]: Refusing update from remote peer cb1c70b29fa44bb4ae71695e534b63ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:24:34.748028 8911 consensus_queue.cc:1035] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7474eb9830654b819e8b7903ede8d3b0" member_type: VOTER last_known_addr { host: "127.7.43.1" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:34.748342 8905 consensus_queue.cc:1035] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8925240c42d343e5a3460b256a5b0fc8" member_type: VOTER last_known_addr { host: "127.7.43.3" port: 44519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:24:34.775964 8108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:34.776225 8108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:24:34.779719 8108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dd1baa1efb348168882885d07ee3c83: Sending DeleteTablet for 3 replicas of tablet b2eb3e8be72949fcb10c7ad4ac97e2eb
I20260810 19:24:34.780984 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:34 UTC) from {username='slave'} at 127.0.0.1:44738
I20260810 19:24:34.780998 8517 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:34 UTC) from {username='slave'} at 127.0.0.1:56892
I20260810 19:24:34.781075 8731 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:24:34 UTC) from {username='slave'} at 127.0.0.1:47242
I20260810 19:24:34.783322 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8205
I20260810 19:24:34.797024 8935 tablet_replica.cc:330] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: stopping tablet replica
I20260810 19:24:34.797750 8935 raft_consensus.cc:2229] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:24:34.798646 8935 raft_consensus.cc:2258] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:34.801273 8935 ts_tablet_manager.cc:1905] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:34.804782 8937 tablet_replica.cc:330] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: stopping tablet replica
I20260810 19:24:34.805428 8936 tablet_replica.cc:330] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0: stopping tablet replica
I20260810 19:24:34.806059 8936 raft_consensus.cc:2229] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:34.806579 8937 raft_consensus.cc:2229] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:24:34.807200 8936 raft_consensus.cc:2258] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 7474eb9830654b819e8b7903ede8d3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:34.807687 8937 raft_consensus.cc:2258] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:24:34.809070 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.784905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.7.43.1:42561)
0810 19:24:34.785095 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.785102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.789506 (+ 4404us) server_negotiation.cc:407] Connection header received
0810 19:24:34.789736 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.789743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.789863 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.789976 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:34.798481 (+ 8505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.799778 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.804238 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.804947 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.805077 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.806931 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:34.806946 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:34.806965 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:34.807818 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:34.807888 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:34.807922 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:34.808042 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.808551 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.808685 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:24:34.809269 8935 ts_tablet_manager.cc:1918] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:34.809486 8935 log.cc:1196] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-1/wal/wals/b2eb3e8be72949fcb10c7ad4ac97e2eb
I20260810 19:24:34.810153 8935 ts_tablet_manager.cc:1939] T b2eb3e8be72949fcb10c7ad4ac97e2eb P cb1c70b29fa44bb4ae71695e534b63ca: Deleting consensus metadata
I20260810 19:24:34.810391 8937 ts_tablet_manager.cc:1905] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:24:34.810673 8400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:34.812070 8086 catalog_manager.cc:4841] TS cb1c70b29fa44bb4ae71695e534b63ca (127.7.43.2:36189): tablet b2eb3e8be72949fcb10c7ad4ac97e2eb (table test-table [id=177c5176f73844d2a23faa515f5e5d80]) successfully deleted
I20260810 19:24:34.817137 8937 ts_tablet_manager.cc:1918] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:24:34.817368 8937 log.cc:1196] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Deleting WAL directory at /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786389541863099-7340-0/minicluster-data/ts-2/wal/wals/b2eb3e8be72949fcb10c7ad4ac97e2eb
I20260810 19:24:34.817881 8937 ts_tablet_manager.cc:1939] T b2eb3e8be72949fcb10c7ad4ac97e2eb P 8925240c42d343e5a3460b256a5b0fc8: Deleting consensus metadata
I20260810 19:24:34.819873 8085 catalog_manager.cc:4841] TS 8925240c42d343e5a3460b256a5b0fc8 (127.7.43.3:44519): tablet b2eb3e8be72949fcb10c7ad4ac97e2eb (table test-table [id=177c5176f73844d2a23faa515f5e5d80]) successfully deleted
I20260810 19:24:34.919636 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8205
I20260810 19:24:34.948159 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8417
W20260810 19:24:34.948159 8086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.43.1:42561 (error 108)
I20260810 19:24:34.950441 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.949717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44754)
0810 19:24:34.950143 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:34.950279 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":220,"threads_started":1}
W20260810 19:24:34.950762 8086 catalog_manager.cc:4584] TS 7474eb9830654b819e8b7903ede8d3b0 (127.7.43.1:42561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2eb3e8be72949fcb10c7ad4ac97e2eb: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
I20260810 19:24:34.956755 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:34.949398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.7.43.2:36189)
0810 19:24:34.949616 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:34.949623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:34.949645 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:24:34.949723 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:34.949729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:34.949854 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:34.949969 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:34.950756 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.951835 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.953683 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:34.954107 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:34.954205 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:34.955354 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:34.955364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:34.955380 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:34.956008 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:34.956055 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:34.956083 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:34.956172 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:34.956324 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 19:24:34.956404 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:24:34.957482 8619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:35.004550 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.004004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44770)
0810 19:24:35.004248 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:35.004352 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 19:24:35.005667 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.005258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44778)
0810 19:24:35.005404 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:35.005480 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 19:24:35.066272 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.065785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44784)
0810 19:24:35.066015 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:35.066113 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 19:24:35.067435 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.067000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44792)
0810 19:24:35.067190 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:35.067265 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 19:24:35.068611 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8417
I20260810 19:24:35.094101 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8634
I20260810 19:24:35.100780 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.094878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.7.43.3:44519)
0810 19:24:35.095072 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:35.095077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:35.095095 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:24:35.095269 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:35.095273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:35.095354 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:35.095434 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:35.095880 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:35.096576 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:35.097963 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:35.098350 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:35.098439 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:35.099511 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:35.099521 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:35.099537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:35.100107 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:35.100155 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:35.100175 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:35.100256 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:35.100386 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 19:24:35.100471 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:24:35.101553 8851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:35.148572 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.147958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44796)
0810 19:24:35.148236 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:35.148365 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 19:24:35.149912 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:35.149369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.1:42561 (local address 127.0.0.1:44800)
0810 19:24:35.149643 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 19:24:35.149730 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 19:24:35.188446 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8634
I20260810 19:24:35.207825 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8060
I20260810 19:24:35.208981 8165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:35.315579 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8060
2026-08-10T19:24:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3038 ms)
[----------] 6 tests from EncryptionPolicyTest (23045 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:35 dist-test-slave-7lbf krb5kdc[8955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:35 dist-test-slave-7lbf krb5kdc[8955](info): set up 2 sockets
Aug 10 19:24:35 dist-test-slave-7lbf krb5kdc[8955](info): commencing operation
krb5kdc: starting...
W20260810 19:24:36.594589 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:36 dist-test-slave-7lbf krb5kdc[8955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:36.936581 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:37.278793 8982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:37.279116 8982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:37.279270 8982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:37.286482 8982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:37.286609 8982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:37.286689 8982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:37.286784 8982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:37.286849 8982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:37.296921 8982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:37.298269 8982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:37.299630 8982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:37.311488 8993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:37.311553 8988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:37.311529 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:37.312695 8982 server_base.cc:1034] running on GCE node
I20260810 19:24:37.313469 8982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:37.314908 8982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:37.316105 8982 hybrid_clock.cc:648] HybridClock initialized: now 1786389877316061 us; error 33 us; skew 500 ppm
I20260810 19:24:37.316471 8982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:37 dist-test-slave-7lbf krb5kdc[8955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:37.319555 8982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:37.320981 8982 webserver.cc:460] Webserver started at http://127.0.0.1:41141/ using document root <none> and password file <none>
I20260810 19:24:37.321719 8982 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:37.321841 8982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:37.322136 8982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:37.325132 8982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "d4fb3e83274f457fb57d785db7936b3a"
format_stamp: "Formatted at 2026-08-10 19:24:37 on dist-test-slave-7lbf"
server_key: "98a9b03e0b32f3b7a522a0b8390c1dc5"
server_key_iv: "62a291620489633f0d03e78769c8a469"
server_key_version: "encryptionkey@0"
I20260810 19:24:37.325945 8982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "d4fb3e83274f457fb57d785db7936b3a"
format_stamp: "Formatted at 2026-08-10 19:24:37 on dist-test-slave-7lbf"
server_key: "98a9b03e0b32f3b7a522a0b8390c1dc5"
server_key_iv: "62a291620489633f0d03e78769c8a469"
server_key_version: "encryptionkey@0"
I20260810 19:24:37.331073 8982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 19:24:37.334529 9004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:37.335840 8982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:24:37.336009 8982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "d4fb3e83274f457fb57d785db7936b3a"
format_stamp: "Formatted at 2026-08-10 19:24:37 on dist-test-slave-7lbf"
server_key: "98a9b03e0b32f3b7a522a0b8390c1dc5"
server_key_iv: "62a291620489633f0d03e78769c8a469"
server_key_version: "encryptionkey@0"
I20260810 19:24:37.336190 8982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:37.366576 8982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:37.370508 8982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:37.370803 8982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:37.389071 8982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44413
I20260810 19:24:37.389096 9093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44413 every 8 connection(s)
I20260810 19:24:37.390513 8982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:37.394369 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:37.397120 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 8982
I20260810 19:24:37.397356 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:37.398063 7340 external_mini_cluster.cc:1428] Setting key b2839a142118d99d8f088a92132637ef
I20260810 19:24:37.407223 9094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a: Bootstrap starting.
I20260810 19:24:37.411084 9094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:37.412469 9094 log.cc:824] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:24:37 dist-test-slave-7lbf krb5kdc[8955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:24:37.415550 9094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a: No bootstrap required, opened a new log
I20260810 19:24:37.422675 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:37.400057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58959 (local address 127.0.0.1:44413)
0810 19:24:37.400846 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:37.400892 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:37.400937 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:24:37.402787 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:37.402812 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:37.403257 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:37.403594 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:37.405439 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:37.407670 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:37.409557 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:37.410249 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:37.410405 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:37.415116 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:37.415125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:37.415141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:37.415178 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:37.418282 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:37.419137 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:37.419144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:37.419150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:37.419325 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:37.419780 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:37.419783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:37.419785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:37.420348 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:37.420662 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:37.421256 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 19:24:37.421619 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":354,"threads_started":1}
I20260810 19:24:37.423509 9094 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fb3e83274f457fb57d785db7936b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44413 } }
I20260810 19:24:37.423877 9094 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:37.423993 9094 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fb3e83274f457fb57d785db7936b3a, State: Initialized, Role: FOLLOWER
I20260810 19:24:37.424583 9094 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fb3e83274f457fb57d785db7936b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44413 } }
I20260810 19:24:37.424841 9094 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:37.425002 9094 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:37.425196 9094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:37.426702 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:37.427662 9094 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fb3e83274f457fb57d785db7936b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44413 } }
I20260810 19:24:37.428141 9094 leader_election.cc:302] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4fb3e83274f457fb57d785db7936b3a; no voters:
I20260810 19:24:37.428727 9094 leader_election.cc:288] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:37.428891 9103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:37.430609 9094 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:37.434396 9103 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [term 1 LEADER]: Becoming Leader. State: Replica: d4fb3e83274f457fb57d785db7936b3a, State: Running, Role: LEADER
I20260810 19:24:37.435139 9103 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fb3e83274f457fb57d785db7936b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44413 } }
I20260810 19:24:37.438206 9105 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4fb3e83274f457fb57d785db7936b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fb3e83274f457fb57d785db7936b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44413 } } }
I20260810 19:24:37.438611 9105 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:37.439154 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4fb3e83274f457fb57d785db7936b3a. Latest consensus state: current_term: 1 leader_uuid: "d4fb3e83274f457fb57d785db7936b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fb3e83274f457fb57d785db7936b3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44413 } } }
I20260810 19:24:37.439451 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a [sys.catalog]: This master's current role is: LEADER
W20260810 19:24:37.453279 9135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:24:37.453446 9135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:24:37.453648 9138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:37.455754 9138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:37.465042 9138 catalog_manager.cc:1347] Generated new cluster ID: 3e5c0f2f7d62491bbe6dbf7e4b89ae1b
I20260810 19:24:37.465163 9138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:37.466821 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:37.448080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33829 (local address 127.0.0.1:44413)
0810 19:24:37.448271 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:37.448278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:37.450021 (+ 1743us) server_negotiation.cc:407] Connection header received
0810 19:24:37.450201 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:37.450204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:37.450273 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:37.450358 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:37.452332 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:37.453545 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:37.455330 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:37.456039 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:37.456175 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:37.459349 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:37.459356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:37.459361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:37.459400 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:37.463230 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:37.464235 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:37.464242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:37.464245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:37.464415 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:37.465043 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:37.465049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:37.465052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:37.465461 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:37.465764 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:37.466242 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 19:24:37.466435 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 19:24:37.477032 9138 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:37.478432 9138 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:37.486287 9138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4fb3e83274f457fb57d785db7936b3a: Generated new TSK 0
I20260810 19:24:37.487085 9138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:37.497805 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 8982
I20260810 19:24:37.499758 9088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:37.588548 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 8982
W20260810 19:24:37.608564 8952 connection.cc:535] client connection to 127.0.0.1:44413 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44413): Connection reset by peer (error 104)
2026-08-10T19:24:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:37 dist-test-slave-7lbf krb5kdc[9153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:37 dist-test-slave-7lbf krb5kdc[9153](info): set up 2 sockets
Aug 10 19:24:37 dist-test-slave-7lbf krb5kdc[9153](info): commencing operation
krb5kdc: starting...
W20260810 19:24:38.851755 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.003s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:39 dist-test-slave-7lbf krb5kdc[9153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:39.175189 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:39.525223 9172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:39.525503 9172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:39.525661 9172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:39.532887 9172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:39.532989 9172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:39.533046 9172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:39.533131 9172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:39.533190 9172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:39.543054 9172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:39.544312 9172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:39.545596 9172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:39.556958 9178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:39.557039 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:39.557085 9179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:39.558338 9172 server_base.cc:1034] running on GCE node
I20260810 19:24:39.559163 9172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:39.560634 9172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:39.561887 9172 hybrid_clock.cc:648] HybridClock initialized: now 1786389879561835 us; error 37 us; skew 500 ppm
I20260810 19:24:39.562254 9172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:39 dist-test-slave-7lbf krb5kdc[9153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:39.569326 9172 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:39.570883 9172 webserver.cc:460] Webserver started at http://127.0.0.1:44259/ using document root <none> and password file <none>
I20260810 19:24:39.571566 9172 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:39.571684 9172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:39.571980 9172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:39.575003 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "1780c29fc8484b9db503177f21fa6740"
format_stamp: "Formatted at 2026-08-10 19:24:39 on dist-test-slave-7lbf"
server_key: "639df65c9f6122f654ce42ff82e1b438"
server_key_iv: "fa37c018e2a5b51cd96fbe3ee897c368"
server_key_version: "encryptionkey@0"
I20260810 19:24:39.575738 9172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "1780c29fc8484b9db503177f21fa6740"
format_stamp: "Formatted at 2026-08-10 19:24:39 on dist-test-slave-7lbf"
server_key: "639df65c9f6122f654ce42ff82e1b438"
server_key_iv: "fa37c018e2a5b51cd96fbe3ee897c368"
server_key_version: "encryptionkey@0"
I20260810 19:24:39.580839 9172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:24:39.584360 9193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:39.585675 9172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:24:39.585834 9172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "1780c29fc8484b9db503177f21fa6740"
format_stamp: "Formatted at 2026-08-10 19:24:39 on dist-test-slave-7lbf"
server_key: "639df65c9f6122f654ce42ff82e1b438"
server_key_iv: "fa37c018e2a5b51cd96fbe3ee897c368"
server_key_version: "encryptionkey@0"
I20260810 19:24:39.586051 9172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:39.607371 9172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:39.611387 9172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:39.611662 9172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:39.629436 9172 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37599
I20260810 19:24:39.629459 9283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37599 every 8 connection(s)
I20260810 19:24:39.630863 9172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:39.634579 9284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:39.638077 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9172
I20260810 19:24:39.638269 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:39.638810 7340 external_mini_cluster.cc:1428] Setting key 49b7dc76b54b08dc7ee468d5a8cb9e12
I20260810 19:24:39.646888 9284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740: Bootstrap starting.
I20260810 19:24:39.650089 9284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:39.651244 9284 log.cc:824] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:39.652092 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:39.640716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56477 (local address 127.0.0.1:37599)
0810 19:24:39.641578 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:39.641597 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:39.641643 (+ 46us) server_negotiation.cc:407] Connection header received
0810 19:24:39.643435 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:39.643459 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:39.643876 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:39.644253 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:39.644842 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:39.646226 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:39.647592 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:39.648274 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:39.648455 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:39.649711 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:39.649719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:39.649733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:39.650315 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:39.650404 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:39.650440 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:39.650613 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:39.650882 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 19:24:39.651066 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":345,"threads_started":1}
I20260810 19:24:39.654109 9284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740: No bootstrap required, opened a new log
I20260810 19:24:39.656002 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:39.662060 9284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1780c29fc8484b9db503177f21fa6740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37599 } }
I20260810 19:24:39.662469 9284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:39.662566 9284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1780c29fc8484b9db503177f21fa6740, State: Initialized, Role: FOLLOWER
I20260810 19:24:39.663265 9284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1780c29fc8484b9db503177f21fa6740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37599 } }
I20260810 19:24:39.663569 9284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:39.663712 9284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:39.663873 9284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:39.666826 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:39.660218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45727 (local address 127.0.0.1:37599)
0810 19:24:39.660367 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:39.660373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:39.660909 (+ 536us) server_negotiation.cc:407] Connection header received
0810 19:24:39.661035 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:39.661039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:39.661097 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:39.661188 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:39.661760 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:39.662513 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:39.664321 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:39.664803 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:39.664950 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:39.666151 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:39.666156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:39.666160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:39.666177 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:39.666214 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:39.666229 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:39.666308 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:39.666460 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 19:24:39.666502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:24:39.666824 9284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1780c29fc8484b9db503177f21fa6740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37599 } }
I20260810 19:24:39.667423 9284 leader_election.cc:302] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1780c29fc8484b9db503177f21fa6740; no voters:
I20260810 19:24:39.668040 9284 leader_election.cc:288] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:39.668154 9305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:39.668797 9305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [term 1 LEADER]: Becoming Leader. State: Replica: 1780c29fc8484b9db503177f21fa6740, State: Running, Role: LEADER
I20260810 19:24:39.669464 9305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1780c29fc8484b9db503177f21fa6740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37599 } }
I20260810 19:24:39.670439 9284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:39.672535 9310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1780c29fc8484b9db503177f21fa6740. Latest consensus state: current_term: 1 leader_uuid: "1780c29fc8484b9db503177f21fa6740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1780c29fc8484b9db503177f21fa6740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37599 } } }
I20260810 19:24:39.672590 9306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1780c29fc8484b9db503177f21fa6740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1780c29fc8484b9db503177f21fa6740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37599 } } }
I20260810 19:24:39.672926 9310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:39.672946 9306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:39.674604 9314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:39.680233 9314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:39.689661 9314 catalog_manager.cc:1347] Generated new cluster ID: 9df38923045d43c982b9bfe2f67716f5
I20260810 19:24:39.689790 9314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:39.699810 9314 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:39.701664 9314 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:39.712380 9314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1780c29fc8484b9db503177f21fa6740: Generated new TSK 0
I20260810 19:24:39.713199 9314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:39.725598 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9172
W20260810 19:24:39.726789 9270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56477
W20260810 19:24:39.727694 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:24:39.727892 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9172
W20260810 19:24:39.750826 9149 connection.cc:535] client connection to 127.0.0.1:37599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37599): Connection reset by peer (error 104)
2026-08-10T19:24:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:39 dist-test-slave-7lbf krb5kdc[9337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:39 dist-test-slave-7lbf krb5kdc[9337](info): set up 2 sockets
Aug 10 19:24:39 dist-test-slave-7lbf krb5kdc[9337](info): commencing operation
krb5kdc: starting...
W20260810 19:24:40.994181 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:41 dist-test-slave-7lbf krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:41.317268 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:41.660295 9358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:41.660600 9358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:41.660733 9358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:41.667968 9358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:41.668102 9358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:41.668180 9358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:41.668262 9358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:41.668318 9358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:41.678471 9358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:41.679836 9358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:41.681452 9358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:41.693560 9370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:41.693610 9368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:41.693615 9367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:41.694514 9358 server_base.cc:1034] running on GCE node
I20260810 19:24:41.695302 9358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:41.696785 9358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:41.697985 9358 hybrid_clock.cc:648] HybridClock initialized: now 1786389881697923 us; error 50 us; skew 500 ppm
I20260810 19:24:41.698340 9358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:41 dist-test-slave-7lbf krb5kdc[9337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:41.701432 9358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:41.703052 9358 webserver.cc:460] Webserver started at http://127.0.0.1:40059/ using document root <none> and password file <none>
I20260810 19:24:41.703755 9358 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:41.703881 9358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:41.704202 9358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:41.707297 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "7ca91712336b44f68d09ae270df76256"
format_stamp: "Formatted at 2026-08-10 19:24:41 on dist-test-slave-7lbf"
server_key: "09c8933ed377fb962def8d923a7a83d8"
server_key_iv: "c4a7516827a72badd2ecb3584a80a808"
server_key_version: "encryptionkey@0"
I20260810 19:24:41.708081 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "7ca91712336b44f68d09ae270df76256"
format_stamp: "Formatted at 2026-08-10 19:24:41 on dist-test-slave-7lbf"
server_key: "09c8933ed377fb962def8d923a7a83d8"
server_key_iv: "c4a7516827a72badd2ecb3584a80a808"
server_key_version: "encryptionkey@0"
I20260810 19:24:41.713426 9358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:24:41.716991 9381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:41.718384 9358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:41.718551 9358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "7ca91712336b44f68d09ae270df76256"
format_stamp: "Formatted at 2026-08-10 19:24:41 on dist-test-slave-7lbf"
server_key: "09c8933ed377fb962def8d923a7a83d8"
server_key_iv: "c4a7516827a72badd2ecb3584a80a808"
server_key_version: "encryptionkey@0"
I20260810 19:24:41.718737 9358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:41.724589 9358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:41.728586 9358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:41.728873 9358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:41.746950 9358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46763
I20260810 19:24:41.746995 9454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46763 every 8 connection(s)
I20260810 19:24:41.748373 9358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:41.749351 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9358
I20260810 19:24:41.749569 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:41.750128 7340 external_mini_cluster.cc:1428] Setting key 23e2b914f95dd1bc07c5a7b81050a9f2
I20260810 19:24:41.753120 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:41.757844 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:41.751967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51303 (local address 127.0.0.1:46763)
0810 19:24:41.752741 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:41.752754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:41.752796 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:24:41.754462 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:41.754487 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:41.754821 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:41.755148 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:41.756238 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:41.756247 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:41.756261 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:41.756275 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:41.756355 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:41.756375 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:41.756544 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:41.756642 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 19:24:41.756831 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
I20260810 19:24:41.761577 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:41.765863 9455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256: Bootstrap starting.
I20260810 19:24:41.769263 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:41.766941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51403 (local address 127.0.0.1:46763)
0810 19:24:41.767132 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:41.767139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:41.767467 (+ 328us) server_negotiation.cc:407] Connection header received
0810 19:24:41.767640 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:41.767644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:41.767696 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:41.767776 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:41.768669 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:41.768674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:41.768677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:41.768691 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:41.768731 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:41.768746 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:41.768832 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:41.768958 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 19:24:41.769004 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:24:41.769819 9455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:41.770926 9455 log.cc:824] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:41.773650 9455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256: No bootstrap required, opened a new log
I20260810 19:24:41.781302 9455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca91712336b44f68d09ae270df76256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46763 } }
I20260810 19:24:41.781736 9455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:41.781850 9455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca91712336b44f68d09ae270df76256, State: Initialized, Role: FOLLOWER
I20260810 19:24:41.782430 9455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca91712336b44f68d09ae270df76256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46763 } }
I20260810 19:24:41.782650 9455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:41.782783 9455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:41.782953 9455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:41.785106 9455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca91712336b44f68d09ae270df76256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46763 } }
I20260810 19:24:41.785689 9455 leader_election.cc:302] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ca91712336b44f68d09ae270df76256; no voters:
I20260810 19:24:41.786296 9455 leader_election.cc:288] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:41.786420 9467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:41.787050 9467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [term 1 LEADER]: Becoming Leader. State: Replica: 7ca91712336b44f68d09ae270df76256, State: Running, Role: LEADER
I20260810 19:24:41.787535 9467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca91712336b44f68d09ae270df76256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46763 } }
I20260810 19:24:41.788323 9455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:41.790223 9472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ca91712336b44f68d09ae270df76256. Latest consensus state: current_term: 1 leader_uuid: "7ca91712336b44f68d09ae270df76256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca91712336b44f68d09ae270df76256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46763 } } }
I20260810 19:24:41.790722 9472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:41.790280 9468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ca91712336b44f68d09ae270df76256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca91712336b44f68d09ae270df76256" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46763 } } }
I20260810 19:24:41.791281 9468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:41.798071 9483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:41.803843 9483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:41.812127 9483 catalog_manager.cc:1347] Generated new cluster ID: fd41442826dd4831aab5d3a0ef01bf00
I20260810 19:24:41.812258 9483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:41.825635 9483 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:41.826040 9483 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:41.833467 9483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ca91712336b44f68d09ae270df76256: Generated new TSK 0
I20260810 19:24:41.834312 9483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:41.842412 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9358
W20260810 19:24:41.843606 9449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51303
W20260810 19:24:41.844278 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:24:41.844414 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9358
2026-08-10T19:24:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:41 dist-test-slave-7lbf krb5kdc[9503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:41 dist-test-slave-7lbf krb5kdc[9503](info): set up 2 sockets
Aug 10 19:24:41 dist-test-slave-7lbf krb5kdc[9503](info): commencing operation
krb5kdc: starting...
W20260810 19:24:43.141163 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.001s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:43 dist-test-slave-7lbf krb5kdc[9503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:43.475221 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:43.820431 9527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:43.820765 9527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:43.820906 9527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:43.828120 9527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:43.828253 9527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:43.828336 9527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:43.828416 9527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:43.828485 9527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:43.838824 9527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:43.840193 9527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:43.841558 9527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:43.853549 9536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:43.853420 9540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:43.853578 9542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:43.854595 9527 server_base.cc:1034] running on GCE node
I20260810 19:24:43.855472 9527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:43.857084 9527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:43.858312 9527 hybrid_clock.cc:648] HybridClock initialized: now 1786389883858260 us; error 43 us; skew 500 ppm
I20260810 19:24:43.858687 9527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:43 dist-test-slave-7lbf krb5kdc[9503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:43.861945 9527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:43.863482 9527 webserver.cc:460] Webserver started at http://127.0.0.1:39603/ using document root <none> and password file <none>
I20260810 19:24:43.864219 9527 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:43.864351 9527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:43.864667 9527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:43.867965 9527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "881cb25614d344f78ed4e48d419fb3e1"
format_stamp: "Formatted at 2026-08-10 19:24:43 on dist-test-slave-7lbf"
server_key: "7c3c3c6eb0377e435ab4222a11a93bce"
server_key_iv: "2c6e11df09bccb3995ffc27348f90466"
server_key_version: "encryptionkey@0"
I20260810 19:24:43.868772 9527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "881cb25614d344f78ed4e48d419fb3e1"
format_stamp: "Formatted at 2026-08-10 19:24:43 on dist-test-slave-7lbf"
server_key: "7c3c3c6eb0377e435ab4222a11a93bce"
server_key_iv: "2c6e11df09bccb3995ffc27348f90466"
server_key_version: "encryptionkey@0"
I20260810 19:24:43.874475 9527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:43.878482 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:43.880057 9527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:43.880216 9527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "881cb25614d344f78ed4e48d419fb3e1"
format_stamp: "Formatted at 2026-08-10 19:24:43 on dist-test-slave-7lbf"
server_key: "7c3c3c6eb0377e435ab4222a11a93bce"
server_key_iv: "2c6e11df09bccb3995ffc27348f90466"
server_key_version: "encryptionkey@0"
I20260810 19:24:43.880406 9527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:43.896175 9527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:43.900484 9527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:43.900733 9527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:43.918496 9527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39561
I20260810 19:24:43.918515 9623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39561 every 8 connection(s)
I20260810 19:24:43.919909 9527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:43.923887 9625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:43.928675 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9527
I20260810 19:24:43.928886 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:43.929358 7340 external_mini_cluster.cc:1428] Setting key 561616449a1d5469709e08003b8311e4
I20260810 19:24:43.935420 9625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1: Bootstrap starting.
I20260810 19:24:43.939255 9625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:43.940411 9625 log.cc:824] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:43.943526 9625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1: No bootstrap required, opened a new log
Aug 10 19:24:43 dist-test-slave-7lbf krb5kdc[9503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:24:43.951535 9625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cb25614d344f78ed4e48d419fb3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260810 19:24:43.951973 9625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:43.952096 9625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cb25614d344f78ed4e48d419fb3e1, State: Initialized, Role: FOLLOWER
I20260810 19:24:43.952725 9625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cb25614d344f78ed4e48d419fb3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260810 19:24:43.952976 9625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:43.953117 9625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:43.953181 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:43.931366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60127 (local address 127.0.0.1:39561)
0810 19:24:43.932209 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:43.932222 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:43.932272 (+ 50us) server_negotiation.cc:407] Connection header received
0810 19:24:43.934139 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:43.934166 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:43.934601 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:43.934984 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:43.936551 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:43.938650 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:43.940085 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:43.941196 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:43.941396 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:43.945621 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:43.945630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:43.945645 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:43.945689 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:43.948784 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:43.949510 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:43.949547 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:43.949557 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:43.949750 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:43.950222 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:43.950227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:43.950230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:43.950847 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:43.951127 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:43.951620 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 19:24:43.952070 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":360,"threads_started":1}
I20260810 19:24:43.953300 9625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:43.956270 9625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cb25614d344f78ed4e48d419fb3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260810 19:24:43.956806 9625 leader_election.cc:302] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881cb25614d344f78ed4e48d419fb3e1; no voters:
I20260810 19:24:43.957163 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:43.957432 9625 leader_election.cc:288] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:43.957590 9631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:43.959378 9625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:43.962181 9631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 881cb25614d344f78ed4e48d419fb3e1, State: Running, Role: LEADER
I20260810 19:24:43.962924 9631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cb25614d344f78ed4e48d419fb3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260810 19:24:43.966976 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881cb25614d344f78ed4e48d419fb3e1. Latest consensus state: current_term: 1 leader_uuid: "881cb25614d344f78ed4e48d419fb3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cb25614d344f78ed4e48d419fb3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } } }
I20260810 19:24:43.968489 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:43.967622 9633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881cb25614d344f78ed4e48d419fb3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cb25614d344f78ed4e48d419fb3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } } }
I20260810 19:24:43.969305 9633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:43.976289 9654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:43.981504 9654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:43.993654 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:43.967899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46417 (local address 127.0.0.1:39561)
0810 19:24:43.968394 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:43.968401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:43.975348 (+ 6947us) server_negotiation.cc:407] Connection header received
0810 19:24:43.976399 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:43.976405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:43.976501 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:43.976623 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:43.978627 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:43.979758 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:43.981544 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:43.982210 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:43.982346 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:43.985849 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:43.985866 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:43.985875 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:43.985922 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:43.989751 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:43.990783 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:43.990789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:43.990793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:43.990971 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:43.991536 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:43.991542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:43.991546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:43.992045 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:43.992306 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:43.992962 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 19:24:43.993211 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260810 19:24:44.000191 9654 catalog_manager.cc:1347] Generated new cluster ID: 098de6d4502e491391d5498bf176366b
I20260810 19:24:44.000313 9654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:44.028101 9654 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:44.029484 9654 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:44.038599 9654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881cb25614d344f78ed4e48d419fb3e1: Generated new TSK 0
I20260810 19:24:44.039484 9654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:44.049149 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9527
I20260810 19:24:44.050251 9616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:44.142995 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9527
2026-08-10T19:24:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:44 dist-test-slave-7lbf krb5kdc[9671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:44 dist-test-slave-7lbf krb5kdc[9671](info): set up 2 sockets
Aug 10 19:24:44 dist-test-slave-7lbf krb5kdc[9671](info): commencing operation
krb5kdc: starting...
W20260810 19:24:45.411731 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:45 dist-test-slave-7lbf krb5kdc[9671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
W20260810 19:24:45.916738 7340 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.0.1: real 0.100s user 0.002s sys 0.040s
I20260810 19:24:45.918176 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:46.510330 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:46.510876 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:46.511070 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:46.522921 9695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:46.523129 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:46.523236 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:46.523322 9695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:46.523388 9695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:46.539664 9695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:46.541939 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:46.544169 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:46.563411 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:46.563494 9706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:46.563438 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:46.564724 9695 server_base.cc:1034] running on GCE node
I20260810 19:24:46.566241 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:46.568874 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:46.570251 9695 hybrid_clock.cc:648] HybridClock initialized: now 1786389886570161 us; error 84 us; skew 500 ppm
I20260810 19:24:46.570884 9695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:46 dist-test-slave-7lbf krb5kdc[9671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:46.596935 9695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:46.599608 9695 webserver.cc:460] Webserver started at http://127.0.0.1:36113/ using document root <none> and password file <none>
I20260810 19:24:46.600832 9695 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:46.601068 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:46.601703 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:46.606938 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "16fe7a21c5cc43dd971842723559626e"
format_stamp: "Formatted at 2026-08-10 19:24:46 on dist-test-slave-7lbf"
server_key: "b697ab5da3b64b7e08cab1b44f4371f4"
server_key_iv: "0e112b17f6955f96b84610359862b0cf"
server_key_version: "encryptionkey@0"
I20260810 19:24:46.608038 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "16fe7a21c5cc43dd971842723559626e"
format_stamp: "Formatted at 2026-08-10 19:24:46 on dist-test-slave-7lbf"
server_key: "b697ab5da3b64b7e08cab1b44f4371f4"
server_key_iv: "0e112b17f6955f96b84610359862b0cf"
server_key_version: "encryptionkey@0"
I20260810 19:24:46.617568 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20260810 19:24:46.623776 9720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:46.626339 9695 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:24:46.626612 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "16fe7a21c5cc43dd971842723559626e"
format_stamp: "Formatted at 2026-08-10 19:24:46 on dist-test-slave-7lbf"
server_key: "b697ab5da3b64b7e08cab1b44f4371f4"
server_key_iv: "0e112b17f6955f96b84610359862b0cf"
server_key_version: "encryptionkey@0"
I20260810 19:24:46.626947 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:46.664321 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:46.671394 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:46.671864 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:46.700811 9695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35149
I20260810 19:24:46.700876 9790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35149 every 8 connection(s)
I20260810 19:24:46.703150 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:46.706278 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9695
I20260810 19:24:46.706554 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:46.707578 7340 external_mini_cluster.cc:1428] Setting key 9cbd8177899c615422e09b9e65695bde
I20260810 19:24:46.710638 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:46.729128 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e: Bootstrap starting.
I20260810 19:24:46.729221 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:46.710626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48195 (local address 127.0.0.1:35149)
0810 19:24:46.711935 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:46.711961 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:46.712013 (+ 52us) server_negotiation.cc:407] Connection header received
0810 19:24:46.714963 (+ 2950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:46.715009 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:46.715741 (+ 732us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:46.716434 (+ 693us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:46.717455 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:46.719612 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:46.721766 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:46.722907 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:46.723293 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:46.725165 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:46.725176 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:46.725200 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:46.726178 (+ 978us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:46.726318 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:46.726374 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:46.726560 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:46.726938 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:24:46.727431 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":545,"threads_started":1}
I20260810 19:24:46.735752 9791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:46.736078 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:46.737499 9791 log.cc:824] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:46.742291 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e: No bootstrap required, opened a new log
I20260810 19:24:46.752409 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:46.742860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45011 (local address 127.0.0.1:35149)
0810 19:24:46.743189 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:46.743207 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:46.743608 (+ 401us) server_negotiation.cc:407] Connection header received
0810 19:24:46.743921 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:46.743935 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:46.744041 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:46.744210 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:46.745119 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:46.746379 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:46.748470 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:46.749272 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:46.749487 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:46.751186 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:46.751193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:46.751207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:46.751234 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:46.751303 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:46.751339 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:46.751479 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:46.751712 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 19:24:46.751914 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 19:24:46.755780 9791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fe7a21c5cc43dd971842723559626e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260810 19:24:46.756392 9791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:46.756505 9791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fe7a21c5cc43dd971842723559626e, State: Initialized, Role: FOLLOWER
I20260810 19:24:46.757324 9791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fe7a21c5cc43dd971842723559626e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260810 19:24:46.757732 9791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:46.757893 9791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:46.758152 9791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:46.762457 9791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fe7a21c5cc43dd971842723559626e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260810 19:24:46.763093 9791 leader_election.cc:302] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fe7a21c5cc43dd971842723559626e; no voters:
I20260810 19:24:46.763696 9791 leader_election.cc:288] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:46.763936 9803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:46.764722 9803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [term 1 LEADER]: Becoming Leader. State: Replica: 16fe7a21c5cc43dd971842723559626e, State: Running, Role: LEADER
I20260810 19:24:46.765266 9803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fe7a21c5cc43dd971842723559626e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260810 19:24:46.766052 9791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:46.768067 9810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fe7a21c5cc43dd971842723559626e. Latest consensus state: current_term: 1 leader_uuid: "16fe7a21c5cc43dd971842723559626e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fe7a21c5cc43dd971842723559626e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } } }
I20260810 19:24:46.768096 9809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16fe7a21c5cc43dd971842723559626e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fe7a21c5cc43dd971842723559626e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } } }
I20260810 19:24:46.768589 9809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:46.769143 9810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:46.770072 9819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:46.775081 9819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:46.785704 9819 catalog_manager.cc:1347] Generated new cluster ID: a7c3eae7725f440d8d9deda3a8e5383f
I20260810 19:24:46.785817 9819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:46.803656 9819 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:46.806759 9819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:46.832248 9819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16fe7a21c5cc43dd971842723559626e: Generated new TSK 0
I20260810 19:24:46.833145 9819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:46.846174 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9695
W20260810 19:24:46.847307 9785 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48195
W20260810 19:24:46.848189 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:24:46.848345 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9695
2026-08-10T19:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2716 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:46 dist-test-slave-7lbf krb5kdc[9842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:46 dist-test-slave-7lbf krb5kdc[9842](info): set up 2 sockets
Aug 10 19:24:46 dist-test-slave-7lbf krb5kdc[9842](info): commencing operation
krb5kdc: starting...
W20260810 19:24:48.112524 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.003s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:48 dist-test-slave-7lbf krb5kdc[9842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:48.445250 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:48.794369 9863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:48.794690 9863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:48.794839 9863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:48.802301 9863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:48.802418 9863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:48.802503 9863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:48.802595 9863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:48.802660 9863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:48.812752 9863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:48.814201 9863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:48.815573 9863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:48.827869 9876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:48.828056 9879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:48.828100 9875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:48.828891 9863 server_base.cc:1034] running on GCE node
I20260810 19:24:48.829792 9863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:48.831385 9863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:48.832597 9863 hybrid_clock.cc:648] HybridClock initialized: now 1786389888832546 us; error 44 us; skew 500 ppm
I20260810 19:24:48.832947 9863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:48 dist-test-slave-7lbf krb5kdc[9842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:48.836196 9863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:48.837810 9863 webserver.cc:460] Webserver started at http://127.0.0.1:43689/ using document root <none> and password file <none>
I20260810 19:24:48.838531 9863 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:48.838651 9863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:48.838955 9863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:48.842355 9863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "ec3aa2fd1a514beabaf0206ce93fb83d"
format_stamp: "Formatted at 2026-08-10 19:24:48 on dist-test-slave-7lbf"
server_key: "d713dd39538aba3bc3690a96201cf81a"
server_key_iv: "fe7c3c3c3cbc466aa67f883555d79285"
server_key_version: "encryptionkey@0"
I20260810 19:24:48.843163 9863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "ec3aa2fd1a514beabaf0206ce93fb83d"
format_stamp: "Formatted at 2026-08-10 19:24:48 on dist-test-slave-7lbf"
server_key: "d713dd39538aba3bc3690a96201cf81a"
server_key_iv: "fe7c3c3c3cbc466aa67f883555d79285"
server_key_version: "encryptionkey@0"
I20260810 19:24:48.848743 9863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260810 19:24:48.852599 9893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:48.854141 9863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:24:48.854319 9863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "ec3aa2fd1a514beabaf0206ce93fb83d"
format_stamp: "Formatted at 2026-08-10 19:24:48 on dist-test-slave-7lbf"
server_key: "d713dd39538aba3bc3690a96201cf81a"
server_key_iv: "fe7c3c3c3cbc466aa67f883555d79285"
server_key_version: "encryptionkey@0"
I20260810 19:24:48.854522 9863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:48.860589 9863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:48.865039 9863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:48.865336 9863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:48.884632 9863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41293
I20260810 19:24:48.884680 9976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41293 every 8 connection(s)
I20260810 19:24:48.886183 9863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:48.889072 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 9863
I20260810 19:24:48.889264 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:48.889870 7340 external_mini_cluster.cc:1428] Setting key fd39f71379a09011e94320bc0a36d230
I20260810 19:24:48.890606 9977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:48.897961 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:48.891800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40309 (local address 127.0.0.1:41293)
0810 19:24:48.892622 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:48.892636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:48.892681 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:24:48.894287 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:48.894314 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:48.894723 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:48.895082 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:48.896179 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:48.896185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:48.896199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:48.896213 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:48.896289 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:48.896313 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:48.896529 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:48.896630 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 19:24:48.896824 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":353,"threads_started":1}
I20260810 19:24:48.901767 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:48.904913 9977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d: Bootstrap starting.
I20260810 19:24:48.908784 9977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:48.909582 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:48.907152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43543 (local address 127.0.0.1:41293)
0810 19:24:48.907342 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:48.907349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:48.907748 (+ 399us) server_negotiation.cc:407] Connection header received
0810 19:24:48.907892 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:48.907897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:48.907968 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:48.908083 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:48.908913 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:48.908918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:48.908922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:48.908939 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:48.908986 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:48.909007 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:48.909125 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:48.909203 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 19:24:48.909266 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 19:24:48.910162 9977 log.cc:824] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:48.912923 9977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d: No bootstrap required, opened a new log
I20260810 19:24:48.920251 9977 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41293 } }
I20260810 19:24:48.920674 9977 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:48.920774 9977 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3aa2fd1a514beabaf0206ce93fb83d, State: Initialized, Role: FOLLOWER
I20260810 19:24:48.921433 9977 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41293 } }
I20260810 19:24:48.921703 9977 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:48.921844 9977 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:48.921999 9977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:48.924165 9977 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41293 } }
I20260810 19:24:48.924656 9977 leader_election.cc:302] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3aa2fd1a514beabaf0206ce93fb83d; no voters:
I20260810 19:24:48.925379 9977 leader_election.cc:288] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:48.925546 9993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:48.926306 9993 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [term 1 LEADER]: Becoming Leader. State: Replica: ec3aa2fd1a514beabaf0206ce93fb83d, State: Running, Role: LEADER
I20260810 19:24:48.926910 9993 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41293 } }
I20260810 19:24:48.927431 9977 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:48.929682 9994 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41293 } } }
I20260810 19:24:48.929765 9996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3aa2fd1a514beabaf0206ce93fb83d. Latest consensus state: current_term: 1 leader_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3aa2fd1a514beabaf0206ce93fb83d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41293 } } }
I20260810 19:24:48.930140 9994 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:48.930164 9996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:48.933099 10002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:48.938896 10002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:48.948779 10002 catalog_manager.cc:1347] Generated new cluster ID: a74c158422c6473d936321065547d436
I20260810 19:24:48.948892 10002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:48.967852 10002 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:48.968319 10002 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:48.978060 10002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec3aa2fd1a514beabaf0206ce93fb83d: Generated new TSK 0
I20260810 19:24:48.979280 10002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:48.995587 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9863
W20260810 19:24:48.996802 9969 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40309
W20260810 19:24:48.997619 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 9863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:24:48.997782 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 9863
2026-08-10T19:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:49 dist-test-slave-7lbf krb5kdc[10030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:49 dist-test-slave-7lbf krb5kdc[10030](info): set up 2 sockets
Aug 10 19:24:49 dist-test-slave-7lbf krb5kdc[10030](info): commencing operation
krb5kdc: starting...
W20260810 19:24:50.268326 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:50 dist-test-slave-7lbf krb5kdc[10030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:50.596221 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:50.948632 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:50.949018 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:50.949173 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:50.956899 10051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:50.957032 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:50.957119 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:50.957213 10051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:50.957267 10051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:50.967245 10051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:50.968576 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:50.969964 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:50.982275 10056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:50.982213 10057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:50.982134 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:50.983446 10051 server_base.cc:1034] running on GCE node
I20260810 19:24:50.984274 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:50.985733 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:50.986936 10051 hybrid_clock.cc:648] HybridClock initialized: now 1786389890986904 us; error 58 us; skew 500 ppm
I20260810 19:24:50.987288 10051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:50 dist-test-slave-7lbf krb5kdc[10030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:50.990490 10051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:50.992063 10051 webserver.cc:460] Webserver started at http://127.0.0.1:36867/ using document root <none> and password file <none>
I20260810 19:24:50.992784 10051 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:50.992903 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:50.993206 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:50.996268 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "2bbcd618fb9343c7ae8371228e90afe6"
format_stamp: "Formatted at 2026-08-10 19:24:50 on dist-test-slave-7lbf"
server_key: "e8cac26c017b47c710e294acfdea3f70"
server_key_iv: "3d810b3b46cdd1b12b73d102414ac36d"
server_key_version: "encryptionkey@0"
I20260810 19:24:50.997076 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "2bbcd618fb9343c7ae8371228e90afe6"
format_stamp: "Formatted at 2026-08-10 19:24:50 on dist-test-slave-7lbf"
server_key: "e8cac26c017b47c710e294acfdea3f70"
server_key_iv: "3d810b3b46cdd1b12b73d102414ac36d"
server_key_version: "encryptionkey@0"
I20260810 19:24:51.002336 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 19:24:51.006059 10068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:51.007579 10051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:24:51.007771 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "2bbcd618fb9343c7ae8371228e90afe6"
format_stamp: "Formatted at 2026-08-10 19:24:50 on dist-test-slave-7lbf"
server_key: "e8cac26c017b47c710e294acfdea3f70"
server_key_iv: "3d810b3b46cdd1b12b73d102414ac36d"
server_key_version: "encryptionkey@0"
I20260810 19:24:51.007995 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:51.051807 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:51.056249 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:51.056528 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:51.074443 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35683
I20260810 19:24:51.074497 10171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35683 every 8 connection(s)
I20260810 19:24:51.075819 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:51.078717 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10051
I20260810 19:24:51.078904 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:51.079466 7340 external_mini_cluster.cc:1428] Setting key c2e0e8462b516ded3ac8be86d7c0155a
I20260810 19:24:51.080084 10172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 19:24:51 dist-test-slave-7lbf krb5kdc[10030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:24:51.094659 10172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6: Bootstrap starting.
I20260810 19:24:51.098886 10172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:51.100169 10172 log.cc:824] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:51.103719 10172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6: No bootstrap required, opened a new log
I20260810 19:24:51.103751 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:51.081273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:44613 (local address 127.0.0.1:35683)
0810 19:24:51.082077 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:51.082089 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:51.082132 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:24:51.083686 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:51.083709 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:51.084125 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:51.084476 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:51.086293 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:51.087671 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:51.089390 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:51.090125 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:51.090252 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:51.095161 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:51.095174 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:51.095195 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:51.095242 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:51.099054 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:51.099869 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:51.099877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:51.099886 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:51.100108 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:51.100526 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:51.100531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:51.100534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:51.101189 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:51.101431 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:51.102036 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 19:24:51.102369 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":336,"threads_started":1}
I20260810 19:24:51.107586 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:51.113404 10172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbcd618fb9343c7ae8371228e90afe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260810 19:24:51.113952 10172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:51.114068 10172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbcd618fb9343c7ae8371228e90afe6, State: Initialized, Role: FOLLOWER
I20260810 19:24:51.114697 10172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbcd618fb9343c7ae8371228e90afe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260810 19:24:51.114918 10172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:51.115054 10172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:51.115206 10172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:51.118018 10172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbcd618fb9343c7ae8371228e90afe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260810 19:24:51.118650 10172 leader_election.cc:302] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bbcd618fb9343c7ae8371228e90afe6; no voters:
I20260810 19:24:51.119347 10172 leader_election.cc:288] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:51.119450 10188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:51.121790 10172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:51.121564 10188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [term 1 LEADER]: Becoming Leader. State: Replica: 2bbcd618fb9343c7ae8371228e90afe6, State: Running, Role: LEADER
I20260810 19:24:51.122876 10188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbcd618fb9343c7ae8371228e90afe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260810 19:24:51.126118 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bbcd618fb9343c7ae8371228e90afe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbcd618fb9343c7ae8371228e90afe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } } }
I20260810 19:24:51.126530 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:51.129279 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:51.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:49187 (local address 127.0.0.1:35683)
0810 19:24:51.113106 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:51.113111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:51.113552 (+ 441us) server_negotiation.cc:407] Connection header received
0810 19:24:51.113697 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:51.113702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:51.113784 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:51.113893 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:51.115402 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:51.116552 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:51.117842 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:51.118527 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:51.118659 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:51.121415 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:51.121421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:51.121427 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:51.121469 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:51.124969 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:51.125991 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:51.125997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:51.126000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:51.126201 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:51.126825 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:51.126830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:51.126833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:51.127221 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:51.127515 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:51.128670 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0810 19:24:51.128881 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:24:51.132681 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbcd618fb9343c7ae8371228e90afe6. Latest consensus state: current_term: 1 leader_uuid: "2bbcd618fb9343c7ae8371228e90afe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbcd618fb9343c7ae8371228e90afe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } } }
I20260810 19:24:51.133065 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:51.134766 10197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:51.140811 10197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:51.149796 10197 catalog_manager.cc:1347] Generated new cluster ID: 9848d487a09242baa12e0aae818673b0
I20260810 19:24:51.149919 10197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:51.163475 10197 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:51.165377 10197 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:51.175731 10197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bbcd618fb9343c7ae8371228e90afe6: Generated new TSK 0
I20260810 19:24:51.176889 10197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:51.182301 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10051
I20260810 19:24:51.183602 10165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:51.273391 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10051
W20260810 19:24:51.296948 10028 connection.cc:535] client connection to 127.0.0.1:35683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35683): Connection reset by peer (error 104)
2026-08-10T19:24:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:51 dist-test-slave-7lbf krb5kdc[10219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:51 dist-test-slave-7lbf krb5kdc[10219](info): set up 2 sockets
Aug 10 19:24:51 dist-test-slave-7lbf krb5kdc[10219](info): commencing operation
krb5kdc: starting...
W20260810 19:24:52.535547 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:52 dist-test-slave-7lbf krb5kdc[10219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:52.874846 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:53.224517 10240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:53.224853 10240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:53.225001 10240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:53.232441 10240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:53.232580 10240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:53.232666 10240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:53.232748 10240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:53.232805 10240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:53.242915 10240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:53.244246 10240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:53.245573 10240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:53.257251 10246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:53.257292 10251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:53.257344 10247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:53.258555 10240 server_base.cc:1034] running on GCE node
I20260810 19:24:53.259410 10240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:53.260951 10240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:53.262158 10240 hybrid_clock.cc:648] HybridClock initialized: now 1786389893262107 us; error 39 us; skew 500 ppm
I20260810 19:24:53.262493 10240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:53 dist-test-slave-7lbf krb5kdc[10219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:53.265733 10240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:53.267354 10240 webserver.cc:460] Webserver started at http://127.0.0.1:41129/ using document root <none> and password file <none>
I20260810 19:24:53.268092 10240 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:53.268217 10240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:53.268543 10240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:53.271734 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "e1d8768af8bd400396ea952f8aa28ecf"
format_stamp: "Formatted at 2026-08-10 19:24:53 on dist-test-slave-7lbf"
server_key: "0c5dae732d9f937244071fdf48c15bcb"
server_key_iv: "d729c6f39e8c5bba87d85866f99a8308"
server_key_version: "encryptionkey@0"
I20260810 19:24:53.272524 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "e1d8768af8bd400396ea952f8aa28ecf"
format_stamp: "Formatted at 2026-08-10 19:24:53 on dist-test-slave-7lbf"
server_key: "0c5dae732d9f937244071fdf48c15bcb"
server_key_iv: "d729c6f39e8c5bba87d85866f99a8308"
server_key_version: "encryptionkey@0"
I20260810 19:24:53.277971 10240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:53.281816 10266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:53.283231 10240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:53.283420 10240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "e1d8768af8bd400396ea952f8aa28ecf"
format_stamp: "Formatted at 2026-08-10 19:24:53 on dist-test-slave-7lbf"
server_key: "0c5dae732d9f937244071fdf48c15bcb"
server_key_iv: "d729c6f39e8c5bba87d85866f99a8308"
server_key_version: "encryptionkey@0"
I20260810 19:24:53.283622 10240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:53.309049 10240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:53.313182 10240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:53.313468 10240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:53.331454 10240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46239
I20260810 19:24:53.331472 10356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46239 every 8 connection(s)
I20260810 19:24:53.332883 10240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:53.336716 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:53.337731 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10240
I20260810 19:24:53.337896 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:53.338331 7340 external_mini_cluster.cc:1428] Setting key 2677845907b5b9586e2d35f562eb71e1
I20260810 19:24:53.351548 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf: Bootstrap starting.
I20260810 19:24:53.353724 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:53.340363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:48353 (local address 127.0.0.1:46239)
0810 19:24:53.341163 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:53.341186 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:53.341240 (+ 54us) server_negotiation.cc:407] Connection header received
0810 19:24:53.343323 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:53.343357 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:53.343931 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:53.344395 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:53.344945 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:53.346843 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:53.348288 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:53.349406 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:53.349616 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:53.350790 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:53.350801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:53.350819 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:53.351590 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:53.351700 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:53.351732 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:53.351895 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:53.352161 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 19:24:53.352361 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":297,"threads_started":1}
I20260810 19:24:53.355459 10358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:53.356709 10358 log.cc:824] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:53.357800 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:53.359406 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf: No bootstrap required, opened a new log
I20260810 19:24:53.368505 10358 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d8768af8bd400396ea952f8aa28ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 19:24:53.369076 10358 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:53.369180 10358 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d8768af8bd400396ea952f8aa28ecf, State: Initialized, Role: FOLLOWER
I20260810 19:24:53.369269 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:53.362230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:58989 (local address 127.0.0.1:46239)
0810 19:24:53.362435 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:53.362440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:53.362744 (+ 304us) server_negotiation.cc:407] Connection header received
0810 19:24:53.362982 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:53.362986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:53.363047 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:53.363124 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:53.363729 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:53.364615 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:53.366513 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:53.367057 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:53.367175 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:53.368409 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:53.368414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:53.368418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:53.368435 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:53.368497 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:53.368513 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:53.368607 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:53.368813 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:24:53.368863 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:24:53.369859 10358 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d8768af8bd400396ea952f8aa28ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 19:24:53.370105 10358 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:53.370247 10358 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:53.370416 10358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:53.372597 10358 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d8768af8bd400396ea952f8aa28ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 19:24:53.373080 10358 leader_election.cc:302] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1d8768af8bd400396ea952f8aa28ecf; no voters:
I20260810 19:24:53.373719 10358 leader_election.cc:288] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:53.373908 10373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:53.374871 10373 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [term 1 LEADER]: Becoming Leader. State: Replica: e1d8768af8bd400396ea952f8aa28ecf, State: Running, Role: LEADER
I20260810 19:24:53.375545 10373 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d8768af8bd400396ea952f8aa28ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 19:24:53.375905 10358 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:53.378695 10375 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1d8768af8bd400396ea952f8aa28ecf. Latest consensus state: current_term: 1 leader_uuid: "e1d8768af8bd400396ea952f8aa28ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d8768af8bd400396ea952f8aa28ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260810 19:24:53.379208 10375 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:53.378682 10374 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1d8768af8bd400396ea952f8aa28ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d8768af8bd400396ea952f8aa28ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260810 19:24:53.379431 10374 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:53.386428 10383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:53.392019 10383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:53.400861 10383 catalog_manager.cc:1347] Generated new cluster ID: 3516b560c56b4187afa876e15df4c179
I20260810 19:24:53.400974 10383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:53.434036 10383 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:53.436091 10383 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:53.444873 10383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1d8768af8bd400396ea952f8aa28ecf: Generated new TSK 0
I20260810 19:24:53.445991 10383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:53.456254 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10240
W20260810 19:24:53.457844 10351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:48353
W20260810 19:24:53.458675 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10240: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:24:53.458818 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10240
W20260810 19:24:53.482366 10216 connection.cc:535] client connection to 127.0.0.1:46239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46239): Connection reset by peer (error 104)
2026-08-10T19:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:53 dist-test-slave-7lbf krb5kdc[10405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:53 dist-test-slave-7lbf krb5kdc[10405](info): set up 2 sockets
Aug 10 19:24:53 dist-test-slave-7lbf krb5kdc[10405](info): commencing operation
krb5kdc: starting...
W20260810 19:24:54.738306 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.004s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:54 dist-test-slave-7lbf krb5kdc[10405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:55.074863 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:55.419545 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:55.419845 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:55.419978 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:55.427248 10426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:55.427361 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:55.427436 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:55.427521 10426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:55.427580 10426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:55.437691 10426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:55.439042 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:55.440373 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:55.452003 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:55.451889 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:55.452164 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:55.453069 10426 server_base.cc:1034] running on GCE node
I20260810 19:24:55.453871 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:55.455279 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:55.456482 10426 hybrid_clock.cc:648] HybridClock initialized: now 1786389895456439 us; error 38 us; skew 500 ppm
I20260810 19:24:55.456830 10426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:55 dist-test-slave-7lbf krb5kdc[10405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:55.459954 10426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:55.461400 10426 webserver.cc:460] Webserver started at http://127.0.0.1:35141/ using document root <none> and password file <none>
I20260810 19:24:55.462100 10426 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:55.462210 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:55.462476 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:55.465407 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "cee1557976a8477298091d2b16097366"
format_stamp: "Formatted at 2026-08-10 19:24:55 on dist-test-slave-7lbf"
server_key: "931cd69ae09cbef79b8ccc7842766b79"
server_key_iv: "91a5c4a91e250489565fc9db19db4411"
server_key_version: "encryptionkey@0"
I20260810 19:24:55.466468 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "cee1557976a8477298091d2b16097366"
format_stamp: "Formatted at 2026-08-10 19:24:55 on dist-test-slave-7lbf"
server_key: "931cd69ae09cbef79b8ccc7842766b79"
server_key_iv: "91a5c4a91e250489565fc9db19db4411"
server_key_version: "encryptionkey@0"
I20260810 19:24:55.473721 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 19:24:55.478821 10449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:55.480358 10426 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:24:55.480512 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "cee1557976a8477298091d2b16097366"
format_stamp: "Formatted at 2026-08-10 19:24:55 on dist-test-slave-7lbf"
server_key: "931cd69ae09cbef79b8ccc7842766b79"
server_key_iv: "91a5c4a91e250489565fc9db19db4411"
server_key_version: "encryptionkey@0"
I20260810 19:24:55.480685 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:55.486348 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:55.490389 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:55.490649 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:55.508464 10426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40459
I20260810 19:24:55.508510 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40459 every 8 connection(s)
I20260810 19:24:55.509887 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:55.513787 10537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:55.518512 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10426
I20260810 19:24:55.518682 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:55.519115 7340 external_mini_cluster.cc:1428] Setting key b936fcb0cab694ddb1a6e652685c4153
I20260810 19:24:55.524600 10537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366: Bootstrap starting.
I20260810 19:24:55.526598 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:55.520803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:42565 (local address 127.0.0.1:40459)
0810 19:24:55.521628 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:55.521640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:55.521683 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:24:55.523381 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:55.523403 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:55.523735 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:55.524033 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:55.525085 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:55.525090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:55.525103 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:55.525120 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:55.525178 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:55.525197 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:55.525354 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:55.525463 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 19:24:55.525682 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":356,"threads_started":1}
I20260810 19:24:55.528194 10537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:55.529382 10537 log.cc:824] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:55.530364 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:55.532545 10537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366: No bootstrap required, opened a new log
I20260810 19:24:55.537868 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:55.535483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:54821 (local address 127.0.0.1:40459)
0810 19:24:55.535660 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:55.535666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:55.536054 (+ 388us) server_negotiation.cc:407] Connection header received
0810 19:24:55.536189 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:55.536194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:55.536254 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:55.536357 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:55.537206 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:55.537212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:55.537217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:55.537236 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:55.537287 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:55.537306 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:55.537428 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:55.537473 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 19:24:55.537550 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:24:55.541327 10537 raft_consensus.cc:348] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee1557976a8477298091d2b16097366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260810 19:24:55.541713 10537 raft_consensus.cc:374] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:55.541790 10537 raft_consensus.cc:729] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee1557976a8477298091d2b16097366, State: Initialized, Role: FOLLOWER
I20260810 19:24:55.542276 10537 consensus_queue.cc:260] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee1557976a8477298091d2b16097366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260810 19:24:55.542500 10537 raft_consensus.cc:388] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:55.542625 10537 raft_consensus.cc:482] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:55.542824 10537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:55.545042 10537 raft_consensus.cc:504] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee1557976a8477298091d2b16097366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260810 19:24:55.545620 10537 leader_election.cc:302] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cee1557976a8477298091d2b16097366; no voters:
I20260810 19:24:55.546185 10537 leader_election.cc:288] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:55.546312 10550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:55.547323 10550 raft_consensus.cc:686] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [term 1 LEADER]: Becoming Leader. State: Replica: cee1557976a8477298091d2b16097366, State: Running, Role: LEADER
I20260810 19:24:55.547811 10550 consensus_queue.cc:237] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee1557976a8477298091d2b16097366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260810 19:24:55.548166 10537 sys_catalog.cc:564] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:55.550386 10552 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee1557976a8477298091d2b16097366. Latest consensus state: current_term: 1 leader_uuid: "cee1557976a8477298091d2b16097366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee1557976a8477298091d2b16097366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } } }
I20260810 19:24:55.550796 10552 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:55.551263 10551 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cee1557976a8477298091d2b16097366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee1557976a8477298091d2b16097366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } } }
I20260810 19:24:55.551555 10551 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:55.551697 10562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:55.561372 10562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:55.571345 10562 catalog_manager.cc:1347] Generated new cluster ID: c0e9efaca0ee4114ae793dc4eeda4ab9
I20260810 19:24:55.571484 10562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:55.585908 10562 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:55.586338 10562 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:55.595697 10562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cee1557976a8477298091d2b16097366: Generated new TSK 0
I20260810 19:24:55.596532 10562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:55.606454 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10426
W20260810 19:24:55.607627 10531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:42565
W20260810 19:24:55.608346 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10426: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:24:55.608479 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10426
2026-08-10T19:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2147 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:55 dist-test-slave-7lbf krb5kdc[10585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:55 dist-test-slave-7lbf krb5kdc[10585](info): set up 2 sockets
Aug 10 19:24:55 dist-test-slave-7lbf krb5kdc[10585](info): commencing operation
krb5kdc: starting...
W20260810 19:24:56.885663 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.003s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:57 dist-test-slave-7lbf krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:57.225260 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:57.566638 10605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:57.566934 10605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:57.567067 10605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:57.574293 10605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:57.574401 10605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:57.574476 10605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:57.574560 10605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:57.574617 10605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:57.584389 10605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:57.585721 10605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:57.587009 10605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:57.598524 10616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:57.598594 10614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:57.598563 10618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:57.599706 10605 server_base.cc:1034] running on GCE node
I20260810 19:24:57.600459 10605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:57.601898 10605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:57.603091 10605 hybrid_clock.cc:648] HybridClock initialized: now 1786389897603039 us; error 42 us; skew 500 ppm
I20260810 19:24:57.603435 10605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:57 dist-test-slave-7lbf krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:57.606546 10605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:57.607998 10605 webserver.cc:460] Webserver started at http://127.0.0.1:35275/ using document root <none> and password file <none>
I20260810 19:24:57.608672 10605 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:57.608783 10605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:57.609072 10605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:57.612102 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "96e3ee599b734b6f8dd874b639f3d4a6"
format_stamp: "Formatted at 2026-08-10 19:24:57 on dist-test-slave-7lbf"
server_key: "32720d4ef18afb6ad6d0c0ada73f32ef"
server_key_iv: "9ba29f506135e9854f513ba7130c0bf4"
server_key_version: "encryptionkey@0"
I20260810 19:24:57.612859 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "96e3ee599b734b6f8dd874b639f3d4a6"
format_stamp: "Formatted at 2026-08-10 19:24:57 on dist-test-slave-7lbf"
server_key: "32720d4ef18afb6ad6d0c0ada73f32ef"
server_key_iv: "9ba29f506135e9854f513ba7130c0bf4"
server_key_version: "encryptionkey@0"
I20260810 19:24:57.617972 10605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:57.621483 10627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:57.622918 10605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:24:57.623085 10605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "96e3ee599b734b6f8dd874b639f3d4a6"
format_stamp: "Formatted at 2026-08-10 19:24:57 on dist-test-slave-7lbf"
server_key: "32720d4ef18afb6ad6d0c0ada73f32ef"
server_key_iv: "9ba29f506135e9854f513ba7130c0bf4"
server_key_version: "encryptionkey@0"
I20260810 19:24:57.623281 10605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:57.637707 10605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:57.641830 10605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:57.642093 10605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:57.660200 10605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43619
I20260810 19:24:57.660224 10712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43619 every 8 connection(s)
I20260810 19:24:57.661713 10605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:57.665431 10714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:57.667490 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10605
I20260810 19:24:57.667646 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:57.668067 7340 external_mini_cluster.cc:1428] Setting key 18582764dba0d140fcfaea878d1518c5
I20260810 19:24:57.677081 10714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6: Bootstrap starting.
I20260810 19:24:57.680821 10714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:57.682175 10714 log.cc:824] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:24:57 dist-test-slave-7lbf krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:24:57.685592 10714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6: No bootstrap required, opened a new log
I20260810 19:24:57.692245 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:57.669876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:43191 (local address 127.0.0.1:43619)
0810 19:24:57.670670 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:57.670682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:57.670724 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:24:57.672350 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:57.672373 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:57.672776 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:57.673097 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:57.674825 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:57.676123 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:57.677913 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:57.678578 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:57.678708 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:57.683791 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:57.683802 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:57.683821 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:57.683873 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:57.687621 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:57.688397 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:57.688407 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:57.688416 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:57.688626 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:57.689033 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:57.689038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:57.689042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:57.689794 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:57.690163 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:57.690472 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 19:24:57.690844 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":309,"threads_started":1}
I20260810 19:24:57.695787 10714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43619 } }
I20260810 19:24:57.696269 10714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:57.696381 10714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e3ee599b734b6f8dd874b639f3d4a6, State: Initialized, Role: FOLLOWER
I20260810 19:24:57.696465 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:57.697034 10714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43619 } }
I20260810 19:24:57.697312 10714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:57.697450 10714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:57.697679 10714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:57.700878 10714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43619 } }
I20260810 19:24:57.701542 10714 leader_election.cc:302] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e3ee599b734b6f8dd874b639f3d4a6; no voters:
I20260810 19:24:57.702203 10714 leader_election.cc:288] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:57.702297 10733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:57.704463 10714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:57.705785 10733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 96e3ee599b734b6f8dd874b639f3d4a6, State: Running, Role: LEADER
I20260810 19:24:57.706306 10733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43619 } }
I20260810 19:24:57.709168 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43619 } } }
I20260810 19:24:57.709635 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:57.710925 10735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e3ee599b734b6f8dd874b639f3d4a6. Latest consensus state: current_term: 1 leader_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e3ee599b734b6f8dd874b639f3d4a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43619 } } }
I20260810 19:24:57.711319 10735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:57.721004 10739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:57.724411 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:57.701596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:33579 (local address 127.0.0.1:43619)
0810 19:24:57.701761 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:57.701767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:57.706271 (+ 4504us) server_negotiation.cc:407] Connection header received
0810 19:24:57.706406 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:57.706416 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:57.706506 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:57.706611 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:57.709324 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:57.710673 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:57.712688 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:57.713410 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:57.713506 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:57.717210 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:57.717218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:57.717224 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:24:57.717267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:57.721088 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:57.722155 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:57.722161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:57.722164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:57.722313 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:24:57.722792 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:24:57.722797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:24:57.722800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:24:57.723147 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:57.723404 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:57.723886 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 19:24:57.724083 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:24:57.728621 10739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:24:57.738471 10739 catalog_manager.cc:1347] Generated new cluster ID: d85ea7619fb54b70b88ed8e2ce194411
I20260810 19:24:57.738582 10739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:24:57.764034 10739 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:24:57.765434 10739 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:24:57.782351 10739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96e3ee599b734b6f8dd874b639f3d4a6: Generated new TSK 0
I20260810 19:24:57.783433 10739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:24:57.796598 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10605
I20260810 19:24:57.797771 10708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:24:57.891027 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10605
W20260810 19:24:57.914949 10581 connection.cc:535] client connection to 127.0.0.1:43619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43619): Connection reset by peer (error 104)
2026-08-10T19:24:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:24:58 dist-test-slave-7lbf krb5kdc[10765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:24:58 dist-test-slave-7lbf krb5kdc[10765](info): set up 2 sockets
Aug 10 19:24:58 dist-test-slave-7lbf krb5kdc[10765](info): commencing operation
krb5kdc: starting...
W20260810 19:24:59.146327 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:24:59 dist-test-slave-7lbf krb5kdc[10765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:24:59.468664 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:24:59.814523 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:24:59.814819 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:24:59.814930 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:24:59.822078 10784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:24:59.822188 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:24:59.822242 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:24:59.822304 10784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:24:59.822347 10784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:24:59.832149 10784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:24:59.833395 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:24:59.834744 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:24:59.846689 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:59.846761 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:24:59.846761 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:24:59.847812 10784 server_base.cc:1034] running on GCE node
I20260810 19:24:59.848680 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:24:59.850144 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:24:59.851356 10784 hybrid_clock.cc:648] HybridClock initialized: now 1786389899851285 us; error 64 us; skew 500 ppm
I20260810 19:24:59.851699 10784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:24:59 dist-test-slave-7lbf krb5kdc[10765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:24:59.854853 10784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:24:59.856262 10784 webserver.cc:460] Webserver started at http://127.0.0.1:35457/ using document root <none> and password file <none>
I20260810 19:24:59.856925 10784 fs_manager.cc:359] Metadata directory not provided
I20260810 19:24:59.857048 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:24:59.857336 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:24:59.860365 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "bf22f042074641998c82ae3ff3b00986"
format_stamp: "Formatted at 2026-08-10 19:24:59 on dist-test-slave-7lbf"
server_key: "31cea54c1198401ac2c415cd6e8b1668"
server_key_iv: "9bcadfa7fb846789868ef93c65c0e1b7"
server_key_version: "encryptionkey@0"
I20260810 19:24:59.861114 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "bf22f042074641998c82ae3ff3b00986"
format_stamp: "Formatted at 2026-08-10 19:24:59 on dist-test-slave-7lbf"
server_key: "31cea54c1198401ac2c415cd6e8b1668"
server_key_iv: "9bcadfa7fb846789868ef93c65c0e1b7"
server_key_version: "encryptionkey@0"
I20260810 19:24:59.866178 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:24:59.869625 10808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:24:59.870927 10784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:24:59.871090 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "bf22f042074641998c82ae3ff3b00986"
format_stamp: "Formatted at 2026-08-10 19:24:59 on dist-test-slave-7lbf"
server_key: "31cea54c1198401ac2c415cd6e8b1668"
server_key_iv: "9bcadfa7fb846789868ef93c65c0e1b7"
server_key_version: "encryptionkey@0"
I20260810 19:24:59.871264 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:24:59.911636 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:24:59.915690 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:24:59.915956 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:24:59.934115 10784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36087
I20260810 19:24:59.934134 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36087 every 8 connection(s)
I20260810 19:24:59.935505 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:24:59.939422 10893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:24:59.942509 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10784
I20260810 19:24:59.942711 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:24:59.943318 7340 external_mini_cluster.cc:1428] Setting key 1be48f663bb26a30e8ee3fe744a13c42
I20260810 19:24:59.952749 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986: Bootstrap starting.
I20260810 19:24:59.956722 10893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986: Neither blocks nor log segments found. Creating new log.
I20260810 19:24:59.957868 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:59.945388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:34119 (local address 127.0.0.1:36087)
0810 19:24:59.946357 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:59.946374 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:59.946743 (+ 369us) server_negotiation.cc:407] Connection header received
0810 19:24:59.948582 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:59.948604 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:59.949012 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:59.949326 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:59.950031 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:59.951361 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:59.953374 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:59.954085 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:59.954214 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:59.955591 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:59.955599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:59.955612 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:59.956133 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:59.956205 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:59.956229 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:59.956348 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:59.956639 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 19:24:59.956810 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":399,"threads_started":1}
I20260810 19:24:59.958001 10893 log.cc:824] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986: Log is configured to *not* fsync() on all Append() calls
I20260810 19:24:59.961328 10893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986: No bootstrap required, opened a new log
I20260810 19:24:59.962103 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:24:59.972146 10893 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf22f042074641998c82ae3ff3b00986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260810 19:24:59.972581 10893 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:24:59.972712 10893 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf22f042074641998c82ae3ff3b00986, State: Initialized, Role: FOLLOWER
I20260810 19:24:59.973392 10893 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf22f042074641998c82ae3ff3b00986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260810 19:24:59.973934 10893 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:24:59.974087 10893 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:24:59.974277 10893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:24:59.974855 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:24:59.967561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:41257 (local address 127.0.0.1:36087)
0810 19:24:59.967749 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:24:59.967754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:24:59.968162 (+ 408us) server_negotiation.cc:407] Connection header received
0810 19:24:59.968366 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:24:59.968369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:24:59.968431 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:24:59.968511 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:24:59.969160 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:59.969941 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:59.971999 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:24:59.972414 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:24:59.972500 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:24:59.973837 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:24:59.973843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:24:59.973846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:24:59.973866 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:24:59.973935 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:24:59.973957 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:24:59.974061 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:24:59.974398 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 19:24:59.974471 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:24:59.977258 10893 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf22f042074641998c82ae3ff3b00986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260810 19:24:59.977859 10893 leader_election.cc:302] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf22f042074641998c82ae3ff3b00986; no voters:
I20260810 19:24:59.978427 10893 leader_election.cc:288] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:24:59.978538 10907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:24:59.979267 10907 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [term 1 LEADER]: Becoming Leader. State: Replica: bf22f042074641998c82ae3ff3b00986, State: Running, Role: LEADER
I20260810 19:24:59.979864 10907 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf22f042074641998c82ae3ff3b00986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260810 19:24:59.980679 10893 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:24:59.982369 10912 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf22f042074641998c82ae3ff3b00986. Latest consensus state: current_term: 1 leader_uuid: "bf22f042074641998c82ae3ff3b00986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf22f042074641998c82ae3ff3b00986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } } }
I20260810 19:24:59.982771 10912 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:59.983592 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf22f042074641998c82ae3ff3b00986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf22f042074641998c82ae3ff3b00986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } } }
I20260810 19:24:59.984194 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986 [sys.catalog]: This master's current role is: LEADER
I20260810 19:24:59.985459 10920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:24:59.990711 10920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:00.006737 10920 catalog_manager.cc:1347] Generated new cluster ID: 4a70e2de02f74e80aebb2d84e91d7554
I20260810 19:25:00.006881 10920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:00.021149 10920 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:00.023018 10920 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:00.034670 10920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf22f042074641998c82ae3ff3b00986: Generated new TSK 0
I20260810 19:25:00.035864 10920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:00.045022 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10784
W20260810 19:25:00.046448 10885 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:34119
W20260810 19:25:00.047559 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10784: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:00.047735 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10784
W20260810 19:25:00.074098 10761 connection.cc:535] client connection to 127.0.0.1:36087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36087): Connection reset by peer (error 104)
2026-08-10T19:25:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2158 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:00 dist-test-slave-7lbf krb5kdc[10948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:00 dist-test-slave-7lbf krb5kdc[10948](info): set up 2 sockets
Aug 10 19:25:00 dist-test-slave-7lbf krb5kdc[10948](info): commencing operation
krb5kdc: starting...
W20260810 19:25:01.320310 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:01 dist-test-slave-7lbf krb5kdc[10948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:25:01.652179 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:01.998149 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:01.998451 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:01.998584 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:02.006258 10969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:02.006395 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:02.006469 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:02.006557 10969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:02.006619 10969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:02.016748 10969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:02.018170 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:02.019523 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:02.031446 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:02.031399 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:02.031569 10978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:02.032126 10969 server_base.cc:1034] running on GCE node
I20260810 19:25:02.032933 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:02.034461 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:02.035671 10969 hybrid_clock.cc:648] HybridClock initialized: now 1786389902035622 us; error 40 us; skew 500 ppm
I20260810 19:25:02.036021 10969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:02 dist-test-slave-7lbf krb5kdc[10948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:02.042317 10969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:25:02.043759 10969 webserver.cc:460] Webserver started at http://127.0.0.1:35021/ using document root <none> and password file <none>
I20260810 19:25:02.044418 10969 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:02.044518 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:02.044771 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:02.047777 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "9813cd57bd344a698507f3993599f249"
format_stamp: "Formatted at 2026-08-10 19:25:02 on dist-test-slave-7lbf"
server_key: "f5bde08d371d9e676ba4b132cb9bf321"
server_key_iv: "8225f5ff8310739fd122f20ff214bb97"
server_key_version: "encryptionkey@0"
I20260810 19:25:02.048511 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "9813cd57bd344a698507f3993599f249"
format_stamp: "Formatted at 2026-08-10 19:25:02 on dist-test-slave-7lbf"
server_key: "f5bde08d371d9e676ba4b132cb9bf321"
server_key_iv: "8225f5ff8310739fd122f20ff214bb97"
server_key_version: "encryptionkey@0"
I20260810 19:25:02.053670 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 19:25:02.057075 10997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:02.058444 10969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:02.058590 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "9813cd57bd344a698507f3993599f249"
format_stamp: "Formatted at 2026-08-10 19:25:02 on dist-test-slave-7lbf"
server_key: "f5bde08d371d9e676ba4b132cb9bf321"
server_key_iv: "8225f5ff8310739fd122f20ff214bb97"
server_key_version: "encryptionkey@0"
I20260810 19:25:02.058772 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:02.064298 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:02.068418 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:02.068689 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:02.085638 10969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36615
I20260810 19:25:02.085705 11082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36615 every 8 connection(s)
I20260810 19:25:02.087037 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:02.090894 11084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:02.092689 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 10969
I20260810 19:25:02.092880 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:02.093318 7340 external_mini_cluster.cc:1428] Setting key df97caa71d37b44d418e9b18e1b1d90b
I20260810 19:25:02.101809 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:02.095162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:53041 (local address 127.0.0.1:36615)
0810 19:25:02.096076 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:02.096092 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:02.096150 (+ 58us) server_negotiation.cc:407] Connection header received
0810 19:25:02.097867 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:02.097897 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:02.098381 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:02.098861 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:02.099795 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:02.099804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:02.099820 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:02.099839 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:02.099917 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:02.099944 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:02.100168 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:02.100299 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 19:25:02.100517 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":337,"threads_started":1}
I20260810 19:25:02.105814 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:02.105894 11084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249: Bootstrap starting.
I20260810 19:25:02.108996 11084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:02.110183 11084 log.cc:824] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:02.112557 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:02.110212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:54483 (local address 127.0.0.1:36615)
0810 19:25:02.110404 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:02.110409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:02.110692 (+ 283us) server_negotiation.cc:407] Connection header received
0810 19:25:02.110860 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:02.110869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:02.110936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:02.111019 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:02.111964 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:02.111969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:02.111973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:02.111991 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:02.112037 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:02.112056 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:02.112170 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:02.112221 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 19:25:02.112279 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:25:02.113662 11084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249: No bootstrap required, opened a new log
I20260810 19:25:02.121176 11084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9813cd57bd344a698507f3993599f249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36615 } }
I20260810 19:25:02.121598 11084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:02.121691 11084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9813cd57bd344a698507f3993599f249, State: Initialized, Role: FOLLOWER
I20260810 19:25:02.122224 11084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9813cd57bd344a698507f3993599f249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36615 } }
I20260810 19:25:02.122435 11084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:02.122535 11084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:02.122653 11084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:02.124802 11084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9813cd57bd344a698507f3993599f249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36615 } }
I20260810 19:25:02.125267 11084 leader_election.cc:302] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9813cd57bd344a698507f3993599f249; no voters:
I20260810 19:25:02.125828 11084 leader_election.cc:288] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:02.125957 11102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:02.126580 11102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [term 1 LEADER]: Becoming Leader. State: Replica: 9813cd57bd344a698507f3993599f249, State: Running, Role: LEADER
I20260810 19:25:02.127120 11102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9813cd57bd344a698507f3993599f249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36615 } }
I20260810 19:25:02.127832 11084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:02.129902 11104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9813cd57bd344a698507f3993599f249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9813cd57bd344a698507f3993599f249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36615 } } }
I20260810 19:25:02.130365 11104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:02.130357 11106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9813cd57bd344a698507f3993599f249. Latest consensus state: current_term: 1 leader_uuid: "9813cd57bd344a698507f3993599f249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9813cd57bd344a698507f3993599f249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36615 } } }
I20260810 19:25:02.130762 11106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:02.132714 11116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:02.137985 11116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:02.147282 11116 catalog_manager.cc:1347] Generated new cluster ID: 35beb5f93c9b4530a8fa12b9aef55cbf
I20260810 19:25:02.147400 11116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:02.169397 11116 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:02.169927 11116 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:02.179076 11116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9813cd57bd344a698507f3993599f249: Generated new TSK 0
I20260810 19:25:02.179924 11116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:02.186090 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10969
W20260810 19:25:02.187161 11073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:53041
W20260810 19:25:02.187830 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 10969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:02.187960 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 10969
2026-08-10T19:25:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2139 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:02 dist-test-slave-7lbf krb5kdc[11145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:02 dist-test-slave-7lbf krb5kdc[11145](info): set up 2 sockets
Aug 10 19:25:02 dist-test-slave-7lbf krb5kdc[11145](info): commencing operation
krb5kdc: starting...
W20260810 19:25:03.448601 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.007s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:03 dist-test-slave-7lbf krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:03.784675 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35511
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:35511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:04.130365 11171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:04.130695 11171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:04.130820 11171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:04.138283 11171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:04.138404 11171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:04.138461 11171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:04.138549 11171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:04.138610 11171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:04.148584 11171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:35511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:04.149958 11171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:04.151294 11171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:04.162714 11181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:04.162751 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:04.163219 11180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:04.163794 11171 server_base.cc:1034] running on GCE node
I20260810 19:25:04.164664 11171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:04.166162 11171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:04.167394 11171 hybrid_clock.cc:648] HybridClock initialized: now 1786389904167350 us; error 35 us; skew 500 ppm
I20260810 19:25:04.167752 11171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:04 dist-test-slave-7lbf krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389904, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:04.170823 11171 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:04.172210 11171 webserver.cc:460] Webserver started at http://127.7.43.62:44601/ using document root <none> and password file <none>
I20260810 19:25:04.172896 11171 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:04.173019 11171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:04.173321 11171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:04.176340 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "0c248f9c7568431387b2535678bf8447"
format_stamp: "Formatted at 2026-08-10 19:25:04 on dist-test-slave-7lbf"
server_key: "a392e5ab691bffda3ca22f4a6c1184bb"
server_key_iv: "cb5a68ab94365214ac4632aa77906d24"
server_key_version: "encryptionkey@0"
I20260810 19:25:04.177093 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "0c248f9c7568431387b2535678bf8447"
format_stamp: "Formatted at 2026-08-10 19:25:04 on dist-test-slave-7lbf"
server_key: "a392e5ab691bffda3ca22f4a6c1184bb"
server_key_iv: "cb5a68ab94365214ac4632aa77906d24"
server_key_version: "encryptionkey@0"
I20260810 19:25:04.182390 11171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:25:04.185925 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:04.187247 11171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:25:04.187441 11171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "0c248f9c7568431387b2535678bf8447"
format_stamp: "Formatted at 2026-08-10 19:25:04 on dist-test-slave-7lbf"
server_key: "a392e5ab691bffda3ca22f4a6c1184bb"
server_key_iv: "cb5a68ab94365214ac4632aa77906d24"
server_key_version: "encryptionkey@0"
I20260810 19:25:04.187660 11171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:04.209256 11171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:04.213315 11171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:04.213644 11171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:04.231633 11171 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:35511
I20260810 19:25:04.231642 11283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:35511 every 8 connection(s)
I20260810 19:25:04.233093 11171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:04.236937 11284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:04.238664 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11171
I20260810 19:25:04.238852 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:04.239326 7340 external_mini_cluster.cc:1428] Setting key 89b8cf814331d5f016880560463bae91
I20260810 19:25:04.249603 11284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447: Bootstrap starting.
Aug 10 19:25:04 dist-test-slave-7lbf krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:04.252929 11284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:04.254032 11284 log.cc:824] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:04.257244 11284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447: No bootstrap required, opened a new log
I20260810 19:25:04.261693 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:04.241283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44279 (local address 127.7.43.62:35511)
0810 19:25:04.242116 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:04.242128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:04.242176 (+ 48us) server_negotiation.cc:407] Connection header received
0810 19:25:04.243808 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:04.243836 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:04.244254 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:04.244627 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:04.246138 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:04.247513 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:04.248845 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:04.249550 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:04.249737 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:04.254004 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:04.254012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:04.254030 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:04.254081 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:04.257373 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:04.258171 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:04.258180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:04.258189 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:04.258365 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:04.258839 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:04.258843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:04.258845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:04.259398 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:04.259745 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:04.260266 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 19:25:04.260600 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":352,"threads_started":1}
I20260810 19:25:04.265297 11284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c248f9c7568431387b2535678bf8447" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35511 } }
I20260810 19:25:04.265650 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:04.265734 11284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:04.265834 11284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c248f9c7568431387b2535678bf8447, State: Initialized, Role: FOLLOWER
I20260810 19:25:04.266376 11284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c248f9c7568431387b2535678bf8447" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35511 } }
I20260810 19:25:04.266603 11284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:04.266731 11284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:04.266878 11284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:04.269178 11284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c248f9c7568431387b2535678bf8447" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35511 } }
I20260810 19:25:04.269732 11284 leader_election.cc:302] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c248f9c7568431387b2535678bf8447; no voters:
I20260810 19:25:04.270452 11284 leader_election.cc:288] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:04.270568 11301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:04.272642 11284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:04.275936 11301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [term 1 LEADER]: Becoming Leader. State: Replica: 0c248f9c7568431387b2535678bf8447, State: Running, Role: LEADER
I20260810 19:25:04.276728 11301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c248f9c7568431387b2535678bf8447" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35511 } }
I20260810 19:25:04.280172 11305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c248f9c7568431387b2535678bf8447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c248f9c7568431387b2535678bf8447" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35511 } } }
I20260810 19:25:04.280241 11307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c248f9c7568431387b2535678bf8447. Latest consensus state: current_term: 1 leader_uuid: "0c248f9c7568431387b2535678bf8447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c248f9c7568431387b2535678bf8447" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35511 } } }
I20260810 19:25:04.280579 11305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:04.280580 11307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:04.289633 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:04.270217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51417 (local address 127.7.43.62:35511)
0810 19:25:04.270403 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:04.270408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:04.271036 (+ 628us) server_negotiation.cc:407] Connection header received
0810 19:25:04.271332 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:04.271339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:04.271421 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:04.271495 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:04.273767 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:04.275157 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:04.276993 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:04.277677 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:04.277809 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:04.280966 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:04.280974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:04.280980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:04.281025 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:04.285092 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:04.286332 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:04.286340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:04.286344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:04.286514 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:04.287115 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:04.287122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:04.287125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:04.287561 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:04.287828 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:04.288997 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0810 19:25:04.289200 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:25:04.291535 11322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:04.296789 11322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:04.306337 11322 catalog_manager.cc:1347] Generated new cluster ID: 3e70c4da0da147c39fe0075cff3a34ff
I20260810 19:25:04.306470 11322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:04.322705 11322 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:04.324054 11322 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:04.331573 11322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c248f9c7568431387b2535678bf8447: Generated new TSK 0
I20260810 19:25:04.332406 11322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:04.346658 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11171
I20260810 19:25:04.348431 11274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:04.441056 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11171
W20260810 19:25:04.463367 11143 connection.cc:535] client connection to 127.7.43.62:35511 recv error: Network error: failed to read from TLS socket (remote: 127.7.43.62:35511): Connection reset by peer (error 104)
2026-08-10T19:25:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:04 dist-test-slave-7lbf krb5kdc[11332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:04 dist-test-slave-7lbf krb5kdc[11332](info): set up 2 sockets
Aug 10 19:25:04 dist-test-slave-7lbf krb5kdc[11332](info): commencing operation
krb5kdc: starting...
W20260810 19:25:05.702471 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:05 dist-test-slave-7lbf krb5kdc[11332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:06.038473 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:33369
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:06.385553 11352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:06.385913 11352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:06.386062 11352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:06.393467 11352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:06.393632 11352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:06.393723 11352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:06.393807 11352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:06.393868 11352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:06.404203 11352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:33369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:06.405556 11352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:06.406888 11352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:06.418663 11361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:06.418866 11362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:06.419148 11367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:06.419831 11352 server_base.cc:1034] running on GCE node
I20260810 19:25:06.420635 11352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:06.422123 11352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:06.423342 11352 hybrid_clock.cc:648] HybridClock initialized: now 1786389906423292 us; error 41 us; skew 500 ppm
I20260810 19:25:06.423709 11352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:06 dist-test-slave-7lbf krb5kdc[11332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389906, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:06.426781 11352 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:06.428247 11352 webserver.cc:460] Webserver started at http://127.7.43.62:46455/ using document root <none> and password file <none>
I20260810 19:25:06.428916 11352 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:06.429034 11352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:06.429323 11352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:06.432322 11352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "49f1f7516914427b8a6f2845cc7ff54f"
format_stamp: "Formatted at 2026-08-10 19:25:06 on dist-test-slave-7lbf"
server_key: "7b34ceb04344af1b6f18bcb25fb5e65b"
server_key_iv: "9682f4145b407e3e87900fc5a3fa9946"
server_key_version: "encryptionkey@0"
I20260810 19:25:06.433076 11352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "49f1f7516914427b8a6f2845cc7ff54f"
format_stamp: "Formatted at 2026-08-10 19:25:06 on dist-test-slave-7lbf"
server_key: "7b34ceb04344af1b6f18bcb25fb5e65b"
server_key_iv: "9682f4145b407e3e87900fc5a3fa9946"
server_key_version: "encryptionkey@0"
I20260810 19:25:06.438350 11352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 19:25:06.441763 11383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:06.443149 11352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:25:06.443313 11352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "49f1f7516914427b8a6f2845cc7ff54f"
format_stamp: "Formatted at 2026-08-10 19:25:06 on dist-test-slave-7lbf"
server_key: "7b34ceb04344af1b6f18bcb25fb5e65b"
server_key_iv: "9682f4145b407e3e87900fc5a3fa9946"
server_key_version: "encryptionkey@0"
I20260810 19:25:06.443516 11352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:06.455938 11352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:06.459964 11352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:06.460223 11352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:06.477753 11352 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:33369
I20260810 19:25:06.477772 11464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:33369 every 8 connection(s)
I20260810 19:25:06.479260 11352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:06.483054 11466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:06.488703 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11352
I20260810 19:25:06.488867 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:06.489282 7340 external_mini_cluster.cc:1428] Setting key 511ee49a696e853145329698759fcc71
I20260810 19:25:06.494095 11466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f: Bootstrap starting.
I20260810 19:25:06.497157 11466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:06.498250 11466 log.cc:824] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:06.501217 11466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f: No bootstrap required, opened a new log
I20260810 19:25:06.502215 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:06.491099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57975 (local address 127.7.43.62:33369)
0810 19:25:06.491936 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:06.491952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:06.491995 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:25:06.493705 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:06.493728 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:06.494135 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:06.494515 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:06.495047 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:06.496363 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:06.497773 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:06.498491 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:06.498658 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:06.499950 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:06.499958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:06.499973 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:06.500487 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:06.500582 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:06.500607 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:06.500731 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:06.500997 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 19:25:06.501157 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":368,"threads_started":1}
I20260810 19:25:06.506964 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:06.510106 11466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1f7516914427b8a6f2845cc7ff54f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33369 } }
I20260810 19:25:06.510458 11466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:06.510571 11466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f1f7516914427b8a6f2845cc7ff54f, State: Initialized, Role: FOLLOWER
I20260810 19:25:06.511240 11466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1f7516914427b8a6f2845cc7ff54f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33369 } }
I20260810 19:25:06.511603 11466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:06.511746 11466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:06.511940 11466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:06.514856 11466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1f7516914427b8a6f2845cc7ff54f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33369 } }
I20260810 19:25:06.515415 11466 leader_election.cc:302] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f1f7516914427b8a6f2845cc7ff54f; no voters:
I20260810 19:25:06.516103 11466 leader_election.cc:288] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:06.516227 11485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:06.518172 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:06.511342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52327 (local address 127.7.43.62:33369)
0810 19:25:06.511529 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:06.511534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:06.511887 (+ 353us) server_negotiation.cc:407] Connection header received
0810 19:25:06.512080 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:06.512085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:06.512164 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:06.512279 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:06.512758 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:06.513915 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:06.515186 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:06.515850 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:06.515989 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:06.517198 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:06.517205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:06.517210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:06.517233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:06.517310 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:06.517331 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:06.517420 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:06.517755 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 19:25:06.517810 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:25:06.518126 11485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [term 1 LEADER]: Becoming Leader. State: Replica: 49f1f7516914427b8a6f2845cc7ff54f, State: Running, Role: LEADER
I20260810 19:25:06.518304 11466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:06.518666 11485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1f7516914427b8a6f2845cc7ff54f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33369 } }
I20260810 19:25:06.523224 11487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f1f7516914427b8a6f2845cc7ff54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1f7516914427b8a6f2845cc7ff54f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33369 } } }
I20260810 19:25:06.523584 11487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:06.524091 11488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f1f7516914427b8a6f2845cc7ff54f. Latest consensus state: current_term: 1 leader_uuid: "49f1f7516914427b8a6f2845cc7ff54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1f7516914427b8a6f2845cc7ff54f" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33369 } } }
I20260810 19:25:06.525023 11488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f [sys.catalog]: This master's current role is: LEADER
W20260810 19:25:06.528584 11505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:25:06.528707 11505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:25:06.528852 11506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:06.530576 11506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:06.538811 11506 catalog_manager.cc:1347] Generated new cluster ID: 1f172bf03b6c4cafa04c0100cb53de6d
I20260810 19:25:06.539018 11506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:06.560338 11506 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:06.562238 11506 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:06.570631 11506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f1f7516914427b8a6f2845cc7ff54f: Generated new TSK 0
I20260810 19:25:06.571645 11506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:06.579751 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11352
W20260810 19:25:06.580960 11457 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57975
W20260810 19:25:06.581928 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11352: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:06.582074 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11352
W20260810 19:25:06.606873 11330 connection.cc:535] client connection to 127.7.43.62:33369 recv error: Network error: failed to read from TLS socket (remote: 127.7.43.62:33369): Connection reset by peer (error 104)
2026-08-10T19:25:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:06 dist-test-slave-7lbf krb5kdc[11519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:06 dist-test-slave-7lbf krb5kdc[11519](info): set up 2 sockets
Aug 10 19:25:06 dist-test-slave-7lbf krb5kdc[11519](info): commencing operation
krb5kdc: starting...
W20260810 19:25:07.856879 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:08 dist-test-slave-7lbf krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:08.188284 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35919
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:35919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:08.537315 11540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:08.537626 11540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:08.537781 11540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:08.545135 11540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:08.545251 11540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:08.545331 11540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:08.545415 11540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:08.545478 11540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:08.555398 11540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:35919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:08.556721 11540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:08.558068 11540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:08.569856 11555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:08.569888 11556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:08.569813 11559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:08.570719 11540 server_base.cc:1034] running on GCE node
I20260810 19:25:08.571480 11540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:08.572890 11540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:08.574107 11540 hybrid_clock.cc:648] HybridClock initialized: now 1786389908574058 us; error 33 us; skew 500 ppm
I20260810 19:25:08.574479 11540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:08 dist-test-slave-7lbf krb5kdc[11519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389908, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:08.581125 11540 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:08.582719 11540 webserver.cc:460] Webserver started at http://127.7.43.62:33781/ using document root <none> and password file <none>
I20260810 19:25:08.583449 11540 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:08.583578 11540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:08.583884 11540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:08.587021 11540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "85fa55014b1a4388a42b93042f8665f8"
format_stamp: "Formatted at 2026-08-10 19:25:08 on dist-test-slave-7lbf"
server_key: "e6329e50676834197a5d48daf0aa970c"
server_key_iv: "5ec7b9677737315d18b3c907b583d4e6"
server_key_version: "encryptionkey@0"
I20260810 19:25:08.587771 11540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "85fa55014b1a4388a42b93042f8665f8"
format_stamp: "Formatted at 2026-08-10 19:25:08 on dist-test-slave-7lbf"
server_key: "e6329e50676834197a5d48daf0aa970c"
server_key_iv: "5ec7b9677737315d18b3c907b583d4e6"
server_key_version: "encryptionkey@0"
I20260810 19:25:08.593312 11540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 19:25:08.596935 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:08.598351 11540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:08.598512 11540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "85fa55014b1a4388a42b93042f8665f8"
format_stamp: "Formatted at 2026-08-10 19:25:08 on dist-test-slave-7lbf"
server_key: "e6329e50676834197a5d48daf0aa970c"
server_key_iv: "5ec7b9677737315d18b3c907b583d4e6"
server_key_version: "encryptionkey@0"
I20260810 19:25:08.598704 11540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:08.604485 11540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:08.608382 11540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:08.608654 11540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:08.626173 11540 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:35919
I20260810 19:25:08.626214 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:35919 every 8 connection(s)
I20260810 19:25:08.627597 11540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:08.631332 11661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:08.631656 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11540
I20260810 19:25:08.631814 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:08.632279 7340 external_mini_cluster.cc:1428] Setting key cc18b47a4d421e33507762f0da80bd26
I20260810 19:25:08.640383 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:08.634028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49777 (local address 127.7.43.62:35919)
0810 19:25:08.634883 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:08.634896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:08.634939 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:25:08.637088 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:08.637112 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:08.637490 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:08.637870 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:08.638894 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:08.638900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:08.638914 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:08.638927 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:08.638984 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:08.639007 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:08.639168 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:08.639270 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 19:25:08.639454 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":378,"threads_started":1}
I20260810 19:25:08.643901 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8: Bootstrap starting.
I20260810 19:25:08.644186 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:08.647708 11661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:08.649046 11661 log.cc:824] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:08.651893 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:08.649389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51235 (local address 127.7.43.62:35919)
0810 19:25:08.649620 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:08.649626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:08.649993 (+ 367us) server_negotiation.cc:407] Connection header received
0810 19:25:08.650133 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:08.650139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:08.650207 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:08.650333 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:08.651305 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:08.651310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:08.651314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:08.651332 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:08.651379 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:08.651397 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:08.651519 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:08.651573 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 19:25:08.651627 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:25:08.652855 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8: No bootstrap required, opened a new log
I20260810 19:25:08.660220 11661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa55014b1a4388a42b93042f8665f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35919 } }
I20260810 19:25:08.660598 11661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:08.660699 11661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85fa55014b1a4388a42b93042f8665f8, State: Initialized, Role: FOLLOWER
I20260810 19:25:08.661278 11661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa55014b1a4388a42b93042f8665f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35919 } }
I20260810 19:25:08.661494 11661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:08.661650 11661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:08.661795 11661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:08.664024 11661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa55014b1a4388a42b93042f8665f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35919 } }
I20260810 19:25:08.664516 11661 leader_election.cc:302] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85fa55014b1a4388a42b93042f8665f8; no voters:
I20260810 19:25:08.665110 11661 leader_election.cc:288] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:08.665233 11676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:08.665899 11676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [term 1 LEADER]: Becoming Leader. State: Replica: 85fa55014b1a4388a42b93042f8665f8, State: Running, Role: LEADER
I20260810 19:25:08.666419 11676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa55014b1a4388a42b93042f8665f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35919 } }
I20260810 19:25:08.667199 11661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:08.669394 11680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85fa55014b1a4388a42b93042f8665f8. Latest consensus state: current_term: 1 leader_uuid: "85fa55014b1a4388a42b93042f8665f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa55014b1a4388a42b93042f8665f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35919 } } }
I20260810 19:25:08.669217 11677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85fa55014b1a4388a42b93042f8665f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa55014b1a4388a42b93042f8665f8" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35919 } } }
I20260810 19:25:08.670042 11677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:08.670099 11680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:08.676944 11687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:08.683079 11687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:08.691773 11687 catalog_manager.cc:1347] Generated new cluster ID: 733194decc474624b0f4e59c2ed3579e
I20260810 19:25:08.691891 11687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:08.727674 11687 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:08.728214 11687 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:08.737986 11687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85fa55014b1a4388a42b93042f8665f8: Generated new TSK 0
I20260810 19:25:08.739043 11687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:08.754631 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11540
W20260810 19:25:08.755734 11652 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49777
W20260810 19:25:08.756450 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11540: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:08.756592 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11540
2026-08-10T19:25:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2179 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:08 dist-test-slave-7lbf krb5kdc[11709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:08 dist-test-slave-7lbf krb5kdc[11709](info): set up 2 sockets
Aug 10 19:25:08 dist-test-slave-7lbf krb5kdc[11709](info): commencing operation
krb5kdc: starting...
W20260810 19:25:10.036126 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:10 dist-test-slave-7lbf krb5kdc[11709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:10.374496 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35593
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:10.724233 11739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:10.724540 11739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:10.724632 11739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:10.732368 11739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:10.732498 11739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:10.732546 11739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:10.732609 11739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:10.732651 11739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:10.742888 11739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:35593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:10.744207 11739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:10.745529 11739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:10.757254 11746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:10.757236 11749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:10.757258 11752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:10.758503 11739 server_base.cc:1034] running on GCE node
I20260810 19:25:10.759305 11739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:10.760762 11739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:10.762012 11739 hybrid_clock.cc:648] HybridClock initialized: now 1786389910761948 us; error 37 us; skew 500 ppm
I20260810 19:25:10.762377 11739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:10 dist-test-slave-7lbf krb5kdc[11709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389910, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:10.765468 11739 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:10.767129 11739 webserver.cc:460] Webserver started at http://127.7.43.62:41103/ using document root <none> and password file <none>
I20260810 19:25:10.767799 11739 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:10.767904 11739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:10.768179 11739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:10.771332 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "26f16dded15f42adb13dbdb71b510b1e"
format_stamp: "Formatted at 2026-08-10 19:25:10 on dist-test-slave-7lbf"
server_key: "bd48a5c55e3e7b9c5dcff72241f7b9c3"
server_key_iv: "d421eea4e47308dded82e28d947d70d2"
server_key_version: "encryptionkey@0"
I20260810 19:25:10.772081 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "26f16dded15f42adb13dbdb71b510b1e"
format_stamp: "Formatted at 2026-08-10 19:25:10 on dist-test-slave-7lbf"
server_key: "bd48a5c55e3e7b9c5dcff72241f7b9c3"
server_key_iv: "d421eea4e47308dded82e28d947d70d2"
server_key_version: "encryptionkey@0"
I20260810 19:25:10.777448 11739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 19:25:10.782438 11763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:10.784446 11739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:25:10.784667 11739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "26f16dded15f42adb13dbdb71b510b1e"
format_stamp: "Formatted at 2026-08-10 19:25:10 on dist-test-slave-7lbf"
server_key: "bd48a5c55e3e7b9c5dcff72241f7b9c3"
server_key_iv: "d421eea4e47308dded82e28d947d70d2"
server_key_version: "encryptionkey@0"
I20260810 19:25:10.784919 11739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:10.811638 11739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:10.817373 11739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:10.817762 11739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:10.838627 11739 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:35593
I20260810 19:25:10.838649 11853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:35593 every 8 connection(s)
I20260810 19:25:10.840565 11739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:10.844724 11855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:10.850500 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11739
I20260810 19:25:10.850728 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:10.851323 7340 external_mini_cluster.cc:1428] Setting key 97628fef741451b677e5dd086bdd93e9
I20260810 19:25:10.856410 11855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e: Bootstrap starting.
I20260810 19:25:10.860193 11855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:10.861603 11855 log.cc:824] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:10.864708 11855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e: No bootstrap required, opened a new log
Aug 10 19:25:10 dist-test-slave-7lbf krb5kdc[11709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:10.873438 11855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f16dded15f42adb13dbdb71b510b1e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35593 } }
I20260810 19:25:10.873922 11855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:10.874027 11855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f16dded15f42adb13dbdb71b510b1e, State: Initialized, Role: FOLLOWER
I20260810 19:25:10.874677 11855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f16dded15f42adb13dbdb71b510b1e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35593 } }
I20260810 19:25:10.874945 11855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:10.875103 11855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:10.875315 11855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:10.878108 11855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f16dded15f42adb13dbdb71b510b1e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35593 } }
I20260810 19:25:10.878747 11855 leader_election.cc:302] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26f16dded15f42adb13dbdb71b510b1e; no voters:
I20260810 19:25:10.879499 11855 leader_election.cc:288] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:10.879630 11865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:10.881162 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:10.853561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49583 (local address 127.7.43.62:35593)
0810 19:25:10.854506 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:10.854521 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:10.854570 (+ 49us) server_negotiation.cc:407] Connection header received
0810 19:25:10.856920 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:10.856950 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:10.857484 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:10.857967 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:10.859899 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:10.861992 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:10.863888 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:10.864918 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:10.865101 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:10.870745 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:10.870763 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:10.870786 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:10.870851 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:10.875170 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:10.876309 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:10.876324 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:10.876341 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:10.876579 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:10.877079 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:10.877086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:10.877090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:10.878003 (+ 913us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:10.878348 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:10.878950 (+ 602us) server_negotiation.cc:299] Negotiation successful
0810 19:25:10.879559 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":415,"threads_started":1}
I20260810 19:25:10.881655 11855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:10.883271 11865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [term 1 LEADER]: Becoming Leader. State: Replica: 26f16dded15f42adb13dbdb71b510b1e, State: Running, Role: LEADER
I20260810 19:25:10.883790 11865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f16dded15f42adb13dbdb71b510b1e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35593 } }
I20260810 19:25:10.886283 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:10.886756 11869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26f16dded15f42adb13dbdb71b510b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f16dded15f42adb13dbdb71b510b1e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35593 } } }
I20260810 19:25:10.887130 11869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:10.888391 11870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26f16dded15f42adb13dbdb71b510b1e. Latest consensus state: current_term: 1 leader_uuid: "26f16dded15f42adb13dbdb71b510b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f16dded15f42adb13dbdb71b510b1e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 35593 } } }
I20260810 19:25:10.888672 11870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:10.889456 11880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:10.901186 11880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:10.936213 11880 catalog_manager.cc:1347] Generated new cluster ID: c7cf6f809d104a4e8ff52095b8e53319
I20260810 19:25:10.936354 11880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:10.953037 11880 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:10.954903 11880 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:10.964986 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:10.928845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41433 (local address 127.7.43.62:35593)
0810 19:25:10.929010 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:10.929017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:10.946250 (+ 17233us) server_negotiation.cc:407] Connection header received
0810 19:25:10.946447 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:10.946452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:10.946538 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:10.946640 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:10.948779 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:10.950019 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:10.953063 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:10.953810 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:10.953933 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:10.957487 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:10.957494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:10.957499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:10.957563 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:10.961311 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:10.962253 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:10.962259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:10.962262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:10.962471 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:10.962979 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:10.962984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:10.962988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:10.963403 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:10.963639 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:10.964290 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 19:25:10.964570 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:25:10.967681 11880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26f16dded15f42adb13dbdb71b510b1e: Generated new TSK 0
I20260810 19:25:10.968818 11880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:10.976419 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11739
I20260810 19:25:10.977771 11842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:11.071635 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11739
2026-08-10T19:25:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:11 dist-test-slave-7lbf krb5kdc[11903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:11 dist-test-slave-7lbf krb5kdc[11903](info): set up 2 sockets
Aug 10 19:25:11 dist-test-slave-7lbf krb5kdc[11903](info): commencing operation
krb5kdc: starting...
W20260810 19:25:12.341890 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:12 dist-test-slave-7lbf krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:12.672493 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:33597
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:13.015308 11924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:13.015643 11924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:13.015789 11924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:13.023268 11924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:13.023398 11924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:13.023479 11924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:13.023566 11924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:13.023626 11924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:13.034294 11924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:13.035665 11924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:13.037042 11924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:13.048769 11933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:13.048733 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:13.048774 11934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:13.049672 11924 server_base.cc:1034] running on GCE node
I20260810 19:25:13.050460 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:13.051879 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:13.053092 11924 hybrid_clock.cc:648] HybridClock initialized: now 1786389913053036 us; error 49 us; skew 500 ppm
I20260810 19:25:13.053435 11924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:13 dist-test-slave-7lbf krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389913, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:13.056628 11924 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:13.058234 11924 webserver.cc:460] Webserver started at http://127.7.43.62:39665/ using document root <none> and password file <none>
I20260810 19:25:13.058905 11924 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:13.059026 11924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:13.059312 11924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:13.062362 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "87923a909d574bc7afcc2755d4fa1348"
format_stamp: "Formatted at 2026-08-10 19:25:13 on dist-test-slave-7lbf"
server_key: "84303405d85f5e7f3485a7e228e6ae4f"
server_key_iv: "5be0ab0679ab88c6dd8866cfb7ea5cfb"
server_key_version: "encryptionkey@0"
I20260810 19:25:13.063114 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "87923a909d574bc7afcc2755d4fa1348"
format_stamp: "Formatted at 2026-08-10 19:25:13 on dist-test-slave-7lbf"
server_key: "84303405d85f5e7f3485a7e228e6ae4f"
server_key_iv: "5be0ab0679ab88c6dd8866cfb7ea5cfb"
server_key_version: "encryptionkey@0"
I20260810 19:25:13.068372 11924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:13.071897 11951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:13.073243 11924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:13.073410 11924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "87923a909d574bc7afcc2755d4fa1348"
format_stamp: "Formatted at 2026-08-10 19:25:13 on dist-test-slave-7lbf"
server_key: "84303405d85f5e7f3485a7e228e6ae4f"
server_key_iv: "5be0ab0679ab88c6dd8866cfb7ea5cfb"
server_key_version: "encryptionkey@0"
I20260810 19:25:13.073642 11924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:13.090409 11924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:13.094470 11924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:13.094738 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:13.112547 11924 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:33597
I20260810 19:25:13.112582 12039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:33597 every 8 connection(s)
I20260810 19:25:13.114027 11924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:13.114686 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 11924
I20260810 19:25:13.114866 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:13.115449 7340 external_mini_cluster.cc:1428] Setting key ae1a1e2ff27574551eaf8dc802cc8465
I20260810 19:25:13.118660 12040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:13.130672 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:13.117308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43135 (local address 127.7.43.62:33597)
0810 19:25:13.118146 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:13.118163 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:13.118215 (+ 52us) server_negotiation.cc:407] Connection header received
0810 19:25:13.120476 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:13.120504 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:13.121020 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:13.121472 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:13.122061 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:13.124095 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:13.125476 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:13.126502 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:13.126676 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:13.127819 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:13.127831 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:13.127849 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:13.128582 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:13.128677 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:13.128708 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:13.128842 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:13.129004 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 19:25:13.129277 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":314,"threads_started":1}
I20260810 19:25:13.132385 12040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348: Bootstrap starting.
I20260810 19:25:13.134339 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:13.135820 12040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:13.137097 12040 log.cc:824] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:13.140312 12040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348: No bootstrap required, opened a new log
I20260810 19:25:13.145725 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:13.139696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56999 (local address 127.7.43.62:33597)
0810 19:25:13.139893 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:13.139900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:13.140401 (+ 501us) server_negotiation.cc:407] Connection header received
0810 19:25:13.140590 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:13.140595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:13.140656 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:13.140753 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:13.141285 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:13.142086 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:13.143380 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:13.143823 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:13.143933 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:13.145014 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:13.145022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:13.145025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:13.145042 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:13.145080 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:13.145095 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:13.145180 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:13.145321 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 19:25:13.145408 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:25:13.148138 12040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87923a909d574bc7afcc2755d4fa1348" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33597 } }
I20260810 19:25:13.148502 12040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:13.148591 12040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87923a909d574bc7afcc2755d4fa1348, State: Initialized, Role: FOLLOWER
I20260810 19:25:13.149103 12040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87923a909d574bc7afcc2755d4fa1348" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33597 } }
I20260810 19:25:13.149319 12040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:13.149449 12040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:13.149670 12040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:13.151990 12040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87923a909d574bc7afcc2755d4fa1348" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33597 } }
I20260810 19:25:13.152489 12040 leader_election.cc:302] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87923a909d574bc7afcc2755d4fa1348; no voters:
I20260810 19:25:13.153043 12040 leader_election.cc:288] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:13.153155 12056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:13.153868 12056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [term 1 LEADER]: Becoming Leader. State: Replica: 87923a909d574bc7afcc2755d4fa1348, State: Running, Role: LEADER
I20260810 19:25:13.154482 12056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87923a909d574bc7afcc2755d4fa1348" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33597 } }
I20260810 19:25:13.154983 12040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:13.156948 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87923a909d574bc7afcc2755d4fa1348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87923a909d574bc7afcc2755d4fa1348" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33597 } } }
I20260810 19:25:13.157245 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:13.157735 12059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87923a909d574bc7afcc2755d4fa1348. Latest consensus state: current_term: 1 leader_uuid: "87923a909d574bc7afcc2755d4fa1348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87923a909d574bc7afcc2755d4fa1348" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 33597 } } }
I20260810 19:25:13.158126 12059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:13.159386 12065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:13.165198 12065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:13.174508 12065 catalog_manager.cc:1347] Generated new cluster ID: 387cabdbb4aa4f14b6ca5be699e94cf4
I20260810 19:25:13.174623 12065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:13.201982 12065 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:13.203359 12065 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:13.218000 12065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87923a909d574bc7afcc2755d4fa1348: Generated new TSK 0
I20260810 19:25:13.218873 12065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:13.237061 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11924
W20260810 19:25:13.238257 12033 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43135
W20260810 19:25:13.239022 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 11924: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:13.239154 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 11924
2026-08-10T19:25:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2166 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:13 dist-test-slave-7lbf krb5kdc[12093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:13 dist-test-slave-7lbf krb5kdc[12093](info): set up 2 sockets
Aug 10 19:25:13 dist-test-slave-7lbf krb5kdc[12093](info): commencing operation
krb5kdc: starting...
W20260810 19:25:14.495618 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:14 dist-test-slave-7lbf krb5kdc[12093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:14.820331 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39001
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:15.157778 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:15.158077 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:15.158215 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:15.165437 12110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:15.165583 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:15.165663 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:15.165750 12110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:15.165819 12110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:15.175684 12110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:15.176939 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:15.178279 12110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:15.189561 12118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:15.189620 12117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:15.189648 12121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:15.190414 12110 server_base.cc:1034] running on GCE node
I20260810 19:25:15.191237 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:15.192619 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:15.193813 12110 hybrid_clock.cc:648] HybridClock initialized: now 1786389915193766 us; error 42 us; skew 500 ppm
I20260810 19:25:15.194168 12110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:15 dist-test-slave-7lbf krb5kdc[12093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389915, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:15.200114 12110 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:15.201637 12110 webserver.cc:460] Webserver started at http://127.7.43.62:39765/ using document root <none> and password file <none>
I20260810 19:25:15.202322 12110 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:15.202440 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:15.202724 12110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:15.205791 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "674d07c4ddbf40e787d5f7e136fa3e9e"
format_stamp: "Formatted at 2026-08-10 19:25:15 on dist-test-slave-7lbf"
server_key: "74a82463a46bc4b0d7e9df97de439c36"
server_key_iv: "266667b36be11db3771e94fb41f58ad8"
server_key_version: "encryptionkey@0"
I20260810 19:25:15.206548 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "674d07c4ddbf40e787d5f7e136fa3e9e"
format_stamp: "Formatted at 2026-08-10 19:25:15 on dist-test-slave-7lbf"
server_key: "74a82463a46bc4b0d7e9df97de439c36"
server_key_iv: "266667b36be11db3771e94fb41f58ad8"
server_key_version: "encryptionkey@0"
I20260810 19:25:15.211768 12110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 19:25:15.215265 12129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:15.216560 12110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:25:15.216727 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "674d07c4ddbf40e787d5f7e136fa3e9e"
format_stamp: "Formatted at 2026-08-10 19:25:15 on dist-test-slave-7lbf"
server_key: "74a82463a46bc4b0d7e9df97de439c36"
server_key_iv: "266667b36be11db3771e94fb41f58ad8"
server_key_version: "encryptionkey@0"
I20260810 19:25:15.216914 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:15.222741 12110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:15.226946 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:15.227237 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:15.244798 12110 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:39001
I20260810 19:25:15.244834 12214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:39001 every 8 connection(s)
I20260810 19:25:15.246236 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:15.248931 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12110
I20260810 19:25:15.249119 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:15.249728 7340 external_mini_cluster.cc:1428] Setting key 5e820e498e41ee9afdc3f5bdf469b61c
I20260810 19:25:15.250377 12215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:15.257050 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:15.251519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41443 (local address 127.7.43.62:39001)
0810 19:25:15.252266 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:15.252278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:15.252319 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:25:15.253829 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:15.253852 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:15.254208 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:15.254538 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:15.255539 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:15.255544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:15.255558 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:15.255570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:15.255643 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:15.255666 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:15.255830 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:15.255952 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 19:25:15.256121 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":312,"threads_started":1}
I20260810 19:25:15.260532 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:15.263917 12215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e: Bootstrap starting.
I20260810 19:25:15.267874 12215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:15.268241 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:15.265843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54861 (local address 127.7.43.62:39001)
0810 19:25:15.266041 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:15.266047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:15.266397 (+ 350us) server_negotiation.cc:407] Connection header received
0810 19:25:15.266563 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:15.266566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:15.266613 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:15.266689 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:15.267606 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:15.267610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:15.267614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:25:15.267627 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:15.267664 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:25:15.267679 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:15.267766 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:15.267912 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:25:15.267963 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 19:25:15.269081 12215 log.cc:824] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:15.271759 12215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e: No bootstrap required, opened a new log
I20260810 19:25:15.278746 12215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39001 } }
I20260810 19:25:15.279115 12215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:25:15.279204 12215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674d07c4ddbf40e787d5f7e136fa3e9e, State: Initialized, Role: FOLLOWER
I20260810 19:25:15.279736 12215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39001 } }
I20260810 19:25:15.279949 12215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:25:15.280087 12215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:25:15.280251 12215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:15.282447 12215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39001 } }
I20260810 19:25:15.282919 12215 leader_election.cc:302] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 674d07c4ddbf40e787d5f7e136fa3e9e; no voters:
I20260810 19:25:15.283622 12215 leader_election.cc:288] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:25:15.283751 12235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:15.284400 12235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [term 1 LEADER]: Becoming Leader. State: Replica: 674d07c4ddbf40e787d5f7e136fa3e9e, State: Running, Role: LEADER
I20260810 19:25:15.284904 12235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39001 } }
I20260810 19:25:15.285604 12215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:15.287762 12236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39001 } } }
I20260810 19:25:15.288215 12236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:15.287788 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674d07c4ddbf40e787d5f7e136fa3e9e. Latest consensus state: current_term: 1 leader_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674d07c4ddbf40e787d5f7e136fa3e9e" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39001 } } }
I20260810 19:25:15.289685 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:15.290557 12248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:15.295102 12248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:15.303495 12248 catalog_manager.cc:1347] Generated new cluster ID: cf128bd91abc4ff8b8f162f82a068544
I20260810 19:25:15.303613 12248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:15.337618 12248 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:15.338088 12248 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:15.349545 12248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 674d07c4ddbf40e787d5f7e136fa3e9e: Generated new TSK 0
I20260810 19:25:15.350392 12248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:15.362636 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12110
W20260810 19:25:15.363785 12206 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41443
W20260810 19:25:15.364524 7340 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12110: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 19:25:15.364657 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12110
2026-08-10T19:25:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2122 ms)
[----------] 18 tests from AuthTokenIssuingTest (40032 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:15 dist-test-slave-7lbf krb5kdc[12270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:15 dist-test-slave-7lbf krb5kdc[12270](info): set up 2 sockets
Aug 10 19:25:15 dist-test-slave-7lbf krb5kdc[12270](info): commencing operation
krb5kdc: starting...
W20260810 19:25:16.619561 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:16 dist-test-slave-7lbf krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:16.958554 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:37297
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:37297,127.7.43.61:44281,127.7.43.60:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:17.317121 12288 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:25:17.317451 12288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:17.317584 12288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:17.317710 12288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:17.324982 12288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:17.325071 12288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:17.325135 12288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:17.325209 12288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:17.325294 12288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:17.335368 12288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:37297,127.7.43.61:44281,127.7.43.60:38505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:37297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:17.336724 12288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:17.338186 12288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:17.349864 12297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:17.349848 12302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:17.349980 12299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:17.350878 12288 server_base.cc:1034] running on GCE node
I20260810 19:25:17.351678 12288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:17.353161 12288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:17.354393 12288 hybrid_clock.cc:648] HybridClock initialized: now 1786389917354321 us; error 34 us; skew 500 ppm
I20260810 19:25:17.354718 12288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:17 dist-test-slave-7lbf krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:17.357873 12288 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:17.359252 12288 webserver.cc:460] Webserver started at http://127.7.43.62:40275/ using document root <none> and password file <none>
I20260810 19:25:17.359912 12288 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:17.360002 12288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:17.360277 12288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:17.363303 12288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "6ef60073e6a142d4aecad0a6f8af03f7"
format_stamp: "Formatted at 2026-08-10 19:25:17 on dist-test-slave-7lbf"
server_key: "8cff397d4b28dc25ec9efa987a099200"
server_key_iv: "86c2820161ec4a3ed5bf0dfe54c221ba"
server_key_version: "encryptionkey@0"
I20260810 19:25:17.364086 12288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "6ef60073e6a142d4aecad0a6f8af03f7"
format_stamp: "Formatted at 2026-08-10 19:25:17 on dist-test-slave-7lbf"
server_key: "8cff397d4b28dc25ec9efa987a099200"
server_key_iv: "86c2820161ec4a3ed5bf0dfe54c221ba"
server_key_version: "encryptionkey@0"
I20260810 19:25:17.369287 12288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 19:25:17.372864 12313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:17.374398 12288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:25:17.374540 12288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "6ef60073e6a142d4aecad0a6f8af03f7"
format_stamp: "Formatted at 2026-08-10 19:25:17 on dist-test-slave-7lbf"
server_key: "8cff397d4b28dc25ec9efa987a099200"
server_key_iv: "86c2820161ec4a3ed5bf0dfe54c221ba"
server_key_version: "encryptionkey@0"
I20260810 19:25:17.374727 12288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:17.399931 12288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:17.404450 12288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:17.404696 12288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:17.423130 12288 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:37297
I20260810 19:25:17.423143 12395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:37297 every 8 connection(s)
I20260810 19:25:17.424530 12288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:17.425755 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12288
I20260810 19:25:17.425984 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:17.426811 7340 external_mini_cluster.cc:1428] Setting key a6d513576102f60fc6b4d0b25023b82a
I20260810 19:25:17.429419 12396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:17.438133 12396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:17 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:17.462342 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.441914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.7.43.62:37297)
0810 19:25:17.442770 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:17.442784 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:17.443358 (+ 574us) server_negotiation.cc:407] Connection header received
0810 19:25:17.445713 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:17.445719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:17.445945 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:17.446110 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:17.447683 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.448846 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.450811 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.451367 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.451507 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:17.455386 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:17.455394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:17.455405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:17.455434 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:17.458119 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:17.458952 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:17.458958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:17.458960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:17.459105 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:17.459554 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:17.459557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:17.459559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:17.460019 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:17.460171 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:17.460934 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 19:25:17.461207 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":624,"thread_start_us":377,"threads_started":1}
I20260810 19:25:17.462392 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.441902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:37297 (local address 127.0.0.1:57192)
0810 19:25:17.443176 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.443219 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:17.445310 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:17.446198 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:17.446205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:17.447026 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:17.447466 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:17.447491 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.449014 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.449021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:17.450677 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:17.450694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.451610 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.451623 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:17.451816 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:17.452491 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:17.452518 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:17.455164 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:17.458265 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:17.458278 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:17.458291 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:17.458814 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:17.459220 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:17.459225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:17.459228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:17.459446 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:17.460157 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:17.460166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:17.460427 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 19:25:17.460702 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 19:25:17.461372 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":161,"thread_start_us":766,"threads_started":1}
WARNING: no policy specified for kudu/127.7.43.61@KRBTEST.COM; defaulting to no policy
I20260810 19:25:17.465389 12396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.61@KRBTEST.COM" created.
I20260810 19:25:17.466486 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.466044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:60550)
0810 19:25:17.466229 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.466360 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260810 19:25:17.466738 12315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
I20260810 19:25:17.468245 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.467905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:60560)
0810 19:25:17.468042 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.468123 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:25:17.468466 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:44281: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab.
Entry for principal kudu/127.7.43.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab.
I20260810 19:25:17.503050 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:44281
--webserver_interface=127.7.43.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab
--principal=kudu/127.7.43.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:37297,127.7.43.61:44281,127.7.43.60:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:25:17.517778 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } attempt: 1
I20260810 19:25:17.519021 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.518580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:60576)
0810 19:25:17.518773 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.518886 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:25:17.520061 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.519706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:60590)
0810 19:25:17.519851 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.519923 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:25:17.520254 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:44281: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
I20260810 19:25:17.588495 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } attempt: 2
I20260810 19:25:17.589788 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.589249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:60598)
0810 19:25:17.589532 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.589641 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:25:17.590847 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.590452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:60608)
0810 19:25:17.590625 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.590698 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:25:17.591053 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:44281: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
I20260810 19:25:17.682299 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } attempt: 3
I20260810 19:25:17.683496 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.683043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37176)
0810 19:25:17.683250 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.683351 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 19:25:17.684497 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.684114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37182)
0810 19:25:17.684264 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.684341 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:25:17.684688 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:44281: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
I20260810 19:25:17.827946 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } attempt: 4
I20260810 19:25:17.829299 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.828711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37192)
0810 19:25:17.829077 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.829168 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260810 19:25:17.830309 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.829945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37196)
0810 19:25:17.830093 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.830175 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 19:25:17.830525 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:44281: Network error: Client connection negotiation failed: client connection to 127.7.43.61:44281: connect: Connection refused (error 111)
W20260810 19:25:17.845137 12409 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:25:17.845489 12409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:17.845623 12409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:17.845742 12409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:17.852969 12409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:17.853062 12409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:17.853152 12409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:17.853216 12409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:17.853318 12409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:17.863221 12409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:37297,127.7.43.61:44281,127.7.43.60:38505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab
--principal=kudu/127.7.43.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:44281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.43.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:17.864630 12409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:17.866040 12409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:17.879849 12414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:17.879812 12415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:17.879947 12418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:17.880561 12409 server_base.cc:1034] running on GCE node
I20260810 19:25:17.881390 12409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:17.882797 12409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:17.884025 12409 hybrid_clock.cc:648] HybridClock initialized: now 1786389917883959 us; error 35 us; skew 500 ppm
I20260810 19:25:17.884357 12409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:17 dist-test-slave-7lbf krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:17.887614 12409 init.cc:377] Logged in from keytab as kudu/127.7.43.61@KRBTEST.COM (short username kudu)
I20260810 19:25:17.889139 12409 webserver.cc:460] Webserver started at http://127.7.43.61:44991/ using document root <none> and password file <none>
I20260810 19:25:17.889911 12409 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:17.890007 12409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:17.890285 12409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:17.893430 12409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data/instance:
uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6"
format_stamp: "Formatted at 2026-08-10 19:25:17 on dist-test-slave-7lbf"
server_key: "e597d809218f2b1a67f11c459e910f48"
server_key_iv: "923cd98716fbfe3f95a3b4c377fd810c"
server_key_version: "encryptionkey@0"
I20260810 19:25:17.894253 12409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal/instance:
uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6"
format_stamp: "Formatted at 2026-08-10 19:25:17 on dist-test-slave-7lbf"
server_key: "e597d809218f2b1a67f11c459e910f48"
server_key_iv: "923cd98716fbfe3f95a3b4c377fd810c"
server_key_version: "encryptionkey@0"
I20260810 19:25:17.900316 12409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 19:25:17.903870 12434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:17.905174 12409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:25:17.905334 12409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal
uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6"
format_stamp: "Formatted at 2026-08-10 19:25:17 on dist-test-slave-7lbf"
server_key: "e597d809218f2b1a67f11c459e910f48"
server_key_iv: "923cd98716fbfe3f95a3b4c377fd810c"
server_key_version: "encryptionkey@0"
I20260810 19:25:17.905570 12409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:17.922271 12409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:17.926291 12409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:17.926532 12409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:17.943941 12409 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.61:44281
I20260810 19:25:17.943990 12515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.61:44281 every 8 connection(s)
I20260810 19:25:17.945243 12409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
I20260810 19:25:17.946331 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12409
I20260810 19:25:17.946514 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-1/wal/instance
I20260810 19:25:17.946938 7340 external_mini_cluster.cc:1428] Setting key cfbdf2230ba501304ddb366fb4bb2562
I20260810 19:25:17.949206 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:17.955897 12517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:17 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:17.975889 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.958613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.7.43.62:37297)
0810 19:25:17.958787 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:17.958792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:17.960098 (+ 1306us) server_negotiation.cc:407] Connection header received
0810 19:25:17.962038 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:17.962044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:17.962122 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:17.962211 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:17.963691 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.964441 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.966022 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.966437 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.966553 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:17.970152 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:17.970158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:17.970163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:17.970203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:17.972753 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:17.973540 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:17.973546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:17.973550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:17.973677 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:17.974112 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:17.974117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:17.974121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:17.974429 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:17.974579 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:17.975329 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 19:25:17.975579 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:25:17.976871 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.959105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:37297 (local address 127.0.0.1:52218)
0810 19:25:17.959943 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.959988 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:17.961758 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:17.962451 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:17.962459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:17.962965 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:17.963517 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:17.963536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.964596 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.964600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:17.965873 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:17.965887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.966659 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.966666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:17.966895 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:17.967540 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:17.967562 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:17.969934 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:17.972945 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:17.972952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:17.972966 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:17.973383 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:17.973797 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:17.973802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:17.973805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:17.973994 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:17.974560 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:17.974568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:17.974953 (+ 385us) client_negotiation.cc:769] Sending connection context
0810 19:25:17.975190 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 19:25:17.975699 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":325,"threads_started":1}
WARNING: no policy specified for kudu/127.7.43.60@KRBTEST.COM; defaulting to no policy
I20260810 19:25:17.978713 12517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.60@KRBTEST.COM" created.
Aug 10 19:25:17 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:25:17.992434 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.979448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37202)
0810 19:25:17.979624 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.979644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:17.979728 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:17.980724 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:17.980728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:17.981149 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:17.981276 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:17.981284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.982692 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.982697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:17.983530 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:17.983541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.984314 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.984320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:17.984491 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:17.985009 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:17.985025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:17.987083 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:17.990121 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:17.990126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:17.990130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:17.990463 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:17.990824 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:17.990829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:17.990833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:17.990958 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:17.991523 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:17.991527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:17.991659 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 19:25:17.991773 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 19:25:17.992034 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:25:17.992513 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.979711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.7.43.61:44281)
0810 19:25:17.980314 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:17.980322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:17.980354 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:25:17.980403 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:17.980410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:17.980585 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:17.980714 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:17.981434 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.982548 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.983656 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:17.984177 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:17.984297 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:17.987271 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:17.987282 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:17.987295 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:17.987325 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:17.989997 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:17.990597 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:17.990604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:17.990606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:17.990723 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:17.991068 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:17.991072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:17.991074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:17.991400 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:17.991603 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:17.991965 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 19:25:17.992174 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":327,"threads_started":1}
I20260810 19:25:17.995290 12517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:17.996289 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.995886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:55978)
0810 19:25:17.996034 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.996153 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:25:17.996562 12437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:17.998040 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:17.997735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:55994)
0810 19:25:17.997865 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:17.997943 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:25:17.998251 12517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab.
Entry for principal kudu/127.7.43.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab.
I20260810 19:25:18.020452 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:38505
--webserver_interface=127.7.43.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab
--principal=kudu/127.7.43.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:37297,127.7.43.61:44281,127.7.43.60:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:25:18.047586 12517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 1
I20260810 19:25:18.048877 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.048379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56004)
0810 19:25:18.048594 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.048696 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 19:25:18.049949 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.049582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56006)
0810 19:25:18.049747 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.049833 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 19:25:18.050163 12517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.118396 12517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 2
I20260810 19:25:18.119606 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.119134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56014)
0810 19:25:18.119340 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.119446 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 19:25:18.120561 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.120218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56024)
0810 19:25:18.120364 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.120439 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:25:18.120757 12517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.129770 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } attempt: 5
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:25:18.143121 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.130708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.7.43.61:44281)
0810 19:25:18.130948 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.130953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.130969 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:25:18.131019 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.131023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.131097 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.131189 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.132025 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.132728 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.133948 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.134424 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.134545 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.137895 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.137900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.137903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.137935 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.140567 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.141223 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.141228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.141230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.141349 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.141752 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.141757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.141760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.142048 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.142182 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.142593 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.142811 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:25:18.143110 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.130460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37218)
0810 19:25:18.130654 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.130674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:18.130789 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:18.131250 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:18.131255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:18.131680 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:18.131875 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.131885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.132881 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.132886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.133796 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.133817 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.134586 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.134591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.134785 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.135402 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:18.135420 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:18.137702 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:18.140711 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.140716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.140721 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.141104 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.141459 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.141464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.141468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.141649 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.142166 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:18.142170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:18.142324 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 19:25:18.142453 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 19:25:18.142741 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:25:18.144289 12396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:18.145298 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.144871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56028)
0810 19:25:18.145025 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.145111 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 19:25:18.146274 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.145907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56036)
0810 19:25:18.146066 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.146153 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:25:18.146533 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.197779 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 1
I20260810 19:25:18.199084 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.198538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56046)
0810 19:25:18.198761 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.198877 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 19:25:18.200070 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.199712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56056)
0810 19:25:18.199853 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.199940 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:25:18.200309 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.211977 12517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 3
I20260810 19:25:18.213143 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.212647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56072)
0810 19:25:18.212874 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.212973 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 19:25:18.214375 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.213900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56082)
0810 19:25:18.214159 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.214250 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260810 19:25:18.214617 12517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.268558 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 2
I20260810 19:25:18.269956 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.269329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56098)
0810 19:25:18.269612 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.269763 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 19:25:18.271005 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.270631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56102)
0810 19:25:18.270782 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.270871 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:25:18.271229 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.357813 12517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 4
I20260810 19:25:18.358919 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.358448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56104)
0810 19:25:18.358681 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.358766 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 19:25:18.359864 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.359552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56116)
0810 19:25:18.359690 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.359756 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:25:18.360107 12517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
W20260810 19:25:18.362151 12526 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:25:18.362471 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:18.362565 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:18.362658 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:18.369902 12526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:18.369992 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:18.370065 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:18.370141 12526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:18.370227 12526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:18.377460 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 3
I20260810 19:25:18.378660 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.378139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56122)
0810 19:25:18.378368 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.378476 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 19:25:18.379650 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.379291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56138)
0810 19:25:18.379461 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.379540 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:25:18.379843 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:38505: Network error: Client connection negotiation failed: client connection to 127.7.43.60:38505: connect: Connection refused (error 111)
I20260810 19:25:18.380667 12526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:37297,127.7.43.61:44281,127.7.43.60:38505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab
--principal=kudu/127.7.43.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.43.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:18.382174 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:18.383482 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:18.395041 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:18.395174 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:18.395339 12535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:18.396121 12526 server_base.cc:1034] running on GCE node
I20260810 19:25:18.396869 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:18.398262 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:18.399483 12526 hybrid_clock.cc:648] HybridClock initialized: now 1786389918399418 us; error 40 us; skew 500 ppm
I20260810 19:25:18.399789 12526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389918, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:18.403014 12526 init.cc:377] Logged in from keytab as kudu/127.7.43.60@KRBTEST.COM (short username kudu)
I20260810 19:25:18.404486 12526 webserver.cc:460] Webserver started at http://127.7.43.60:40711/ using document root <none> and password file <none>
I20260810 19:25:18.405151 12526 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:18.405256 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:18.405565 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:18.408604 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data/instance:
uuid: "a8eaac23ac3d4df5a69e51c143f56c1a"
format_stamp: "Formatted at 2026-08-10 19:25:18 on dist-test-slave-7lbf"
server_key: "733608bdff4d0c8c58f3a88fd6115fb4"
server_key_iv: "b58919f7ce626aa9bab495951375b029"
server_key_version: "encryptionkey@0"
I20260810 19:25:18.409373 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal/instance:
uuid: "a8eaac23ac3d4df5a69e51c143f56c1a"
format_stamp: "Formatted at 2026-08-10 19:25:18 on dist-test-slave-7lbf"
server_key: "733608bdff4d0c8c58f3a88fd6115fb4"
server_key_iv: "b58919f7ce626aa9bab495951375b029"
server_key_version: "encryptionkey@0"
I20260810 19:25:18.414534 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 19:25:18.417949 12547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:18.419327 12526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 19:25:18.419486 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal
uuid: "a8eaac23ac3d4df5a69e51c143f56c1a"
format_stamp: "Formatted at 2026-08-10 19:25:18 on dist-test-slave-7lbf"
server_key: "733608bdff4d0c8c58f3a88fd6115fb4"
server_key_iv: "b58919f7ce626aa9bab495951375b029"
server_key_version: "encryptionkey@0"
I20260810 19:25:18.419695 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:18.449116 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:18.453198 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:18.453461 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:18.470903 12526 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.60:38505
I20260810 19:25:18.470925 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.60:38505 every 8 connection(s)
I20260810 19:25:18.472278 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
I20260810 19:25:18.473745 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12526
I20260810 19:25:18.473928 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786389541863099-7340-0/minicluster-data/master-2/wal/instance
I20260810 19:25:18.474493 7340 external_mini_cluster.cc:1428] Setting key 591c2297d56726a672d982a5fc3b759e
I20260810 19:25:18.476912 12634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:18.485745 12634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389918, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:18.498669 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.476524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.7.43.60:38505)
0810 19:25:18.477322 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.477338 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.477389 (+ 51us) server_negotiation.cc:407] Connection header received
0810 19:25:18.479648 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.479676 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.480200 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.480641 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.481999 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.483972 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.485180 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.486211 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.486391 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.490925 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.490937 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.490952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.490992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.494336 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.495183 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.495192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.495200 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.495394 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.495801 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.495804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.495806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.496314 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.496605 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.496999 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.497428 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":322,"threads_started":1}
I20260810 19:25:18.503720 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.488364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.7.43.62:37297)
0810 19:25:18.488942 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.488948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.489382 (+ 434us) server_negotiation.cc:407] Connection header received
0810 19:25:18.489575 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.489581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.489645 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.489750 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.491150 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.492296 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.493736 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.494396 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.494517 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.498134 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.498139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.498142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.498174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.500830 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.501630 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.501634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.501636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.501823 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.502205 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.502209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.502213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.502606 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.502719 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.503257 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.503418 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":434,"thread_start_us":264,"threads_started":1}
I20260810 19:25:18.503916 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.488322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:37297 (local address 127.0.0.1:52234)
0810 19:25:18.489241 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.489274 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:18.489432 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:18.489825 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:18.489832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:18.490640 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:18.490982 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.490996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.492453 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.492457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.493566 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.493580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.494581 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.494587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.494761 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.495358 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:18.495422 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:18.497957 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:18.500951 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.500957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.500961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.501456 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.501947 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.501951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.501953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.502091 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.502744 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:18.502752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:18.502955 (+ 203us) client_negotiation.cc:769] Sending connection context
0810 19:25:18.503191 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 19:25:18.503530 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"spinlock_wait_cycles":12672,"thread_start_us":573,"threads_started":1}
I20260810 19:25:18.505363 12634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389918, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:25:18.516587 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.501459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.7.43.62:37297)
0810 19:25:18.502081 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.502084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.502101 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:25:18.502138 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.502142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.502208 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.502303 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.503783 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.504809 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.506172 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.506757 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.506901 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.511293 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.511298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.511301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.511331 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.513963 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.514758 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.514763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.514767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.514913 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.515283 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.515286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.515288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.515544 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.515732 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.516084 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.516242 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":506,"thread_start_us":306,"threads_started":1}
I20260810 19:25:18.518800 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.506004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:44281 (local address 127.0.0.1:37224)
0810 19:25:18.506172 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.506190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:18.506277 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:18.506542 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:18.506547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:18.506984 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:18.507140 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.507149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.508457 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.508461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.509372 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.509384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.510206 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.510213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.510438 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.510947 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:18.510969 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:18.513025 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:18.516411 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.516415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.516419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.516868 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.517226 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.517231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.517234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.517370 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.517945 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:18.517949 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:18.518079 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 19:25:18.518192 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 19:25:18.518453 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:25:18.518867 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.506144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.7.43.61:44281)
0810 19:25:18.506302 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.506306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.506320 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:25:18.506367 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.506371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.506425 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.506505 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.507294 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.508280 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.509509 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.510055 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.510148 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.513208 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.513213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.513216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.513248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.516279 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.516977 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.516983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.516985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.517123 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.517472 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.517476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.517478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.517825 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.517962 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.518297 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.518490 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:25:18.519799 12634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389918, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
I20260810 19:25:18.534636 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.520475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56144)
0810 19:25:18.520646 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.520664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:18.520755 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:18.521327 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:18.521332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:18.521765 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:18.521938 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.521949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.523349 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.523355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.524636 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.524651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.525680 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.525687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.525918 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.526498 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:18.526520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:18.528459 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:18.532039 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.532044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.532048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.532533 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.532914 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.532918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.532922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.533045 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.533682 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:18.533686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:18.533823 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 19:25:18.533974 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 19:25:18.534247 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 19:25:18.534778 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.520887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.7.43.60:38505)
0810 19:25:18.521040 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.521046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.521065 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:25:18.521112 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.521117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.521182 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.521278 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.522089 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.523209 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.524796 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.525459 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.525648 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.528616 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.528621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.528624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.528650 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.531884 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.532654 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.532660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.532663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.532815 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.533149 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.533153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.533156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.533472 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.533635 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.534077 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.534414 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":51}
I20260810 19:25:18.535035 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.518321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.7.43.61:44281)
0810 19:25:18.519076 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.519081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.519096 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:25:18.519138 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.519142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.519202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.519286 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.520783 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.521716 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.522949 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.523460 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.523582 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.529204 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.529211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.529216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.529254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.531871 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.532513 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.532517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.532520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.532665 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.533140 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.533145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.533148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.533471 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.533837 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.534560 (+ 723us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.534714 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":15592,"threads_started":1}
I20260810 19:25:18.539846 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:18.539913 12634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: Bootstrap starting.
I20260810 19:25:18.543171 12634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:18.544533 12634 log.cc:824] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:18.542510 12352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ef60073e6a142d4aecad0a6f8af03f7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52238
I20260810 19:25:18.547742 12634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: No bootstrap required, opened a new log
I20260810 19:25:18.554700 12634 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.555019 12634 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8eaac23ac3d4df5a69e51c143f56c1a, State: Initialized, Role: FOLLOWER
I20260810 19:25:18.555454 12634 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.556325 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.556715 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:18.557046 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 4
I20260810 19:25:18.557252 12634 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
W20260810 19:25:18.569619 12677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:25:18.569756 12677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:25:18.573180 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.557850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.7.43.60:38505)
0810 19:25:18.558045 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.558050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.558271 (+ 221us) server_negotiation.cc:407] Connection header received
0810 19:25:18.558333 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.558339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.558412 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.558502 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.559360 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.560203 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.561632 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.562263 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.562380 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.566109 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.566116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.566121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.566160 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.569871 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.570669 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.570676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.570683 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.570844 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.571341 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.571373 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.571380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.571816 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.571991 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.572565 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.572790 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:25:18.573415 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.557829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56154)
0810 19:25:18.558018 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.558039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:18.558169 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:18.558535 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:18.558539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:18.559049 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:18.559209 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.559219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.560343 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.560349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.561448 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.561463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.562430 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.562436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.562641 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.563285 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:18.563311 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:18.565925 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:18.570012 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.570018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.570022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.570527 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.570955 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.570960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.570963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.571129 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.571956 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:18.571961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:18.572219 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 19:25:18.572453 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 19:25:18.572868 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:25:18.567358 12526 master_runner.cc:186] Error getting master registration for 127.7.43.62:37297: OK, instance_id { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" instance_seqno: 1786389917417928 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:25:18.578218 12526 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:25:18.578984 12396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: Bootstrap starting.
I20260810 19:25:18.582095 12396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:18.583102 12396 log.cc:824] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:18.585731 12396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: No bootstrap required, opened a new log
I20260810 19:25:18.590698 12396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.591004 12396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef60073e6a142d4aecad0a6f8af03f7, State: Initialized, Role: FOLLOWER
I20260810 19:25:18.591413 12396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.592259 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.592648 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:18.593348 12396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:18.593585 12352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 19:25:18.593830 12352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:18.597563 12352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.600229 12591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a"
I20260810 19:25:18.600775 12591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:18.604208 12591 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef60073e6a142d4aecad0a6f8af03f7 in term 1.
I20260810 19:25:18.604918 12315 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef60073e6a142d4aecad0a6f8af03f7, a8eaac23ac3d4df5a69e51c143f56c1a; no voters:
I20260810 19:25:18.605445 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 19:25:18.611258 12703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:25:18.611378 12703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:25:18.599310 12352 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [CANDIDATE]: Term 1 election: Requested vote from peers 8bc2cf01bd254b7ebfe7b19a1d54a0f6 (127.7.43.61:44281), a8eaac23ac3d4df5a69e51c143f56c1a (127.7.43.60:38505)
I20260810 19:25:18.620433 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef60073e6a142d4aecad0a6f8af03f7, State: Running, Role: LEADER
I20260810 19:25:18.621066 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.604205 12288 master_runner.cc:186] Error getting master registration for 127.7.43.61:44281: OK, instance_id { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" instance_seqno: 1786389917939018 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:25:18.626014 12288 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:25:18.634886 12706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef60073e6a142d4aecad0a6f8af03f7. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.635371 12706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:18.650274 12716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:18.652741 12716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 19:25:18.658402 12320 tablet.cc:2369] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:18.659294 12517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } attempt: 5
I20260810 19:25:18.670154 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.645393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.7.43.61:44281)
0810 19:25:18.645607 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.645614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.651361 (+ 5747us) server_negotiation.cc:407] Connection header received
0810 19:25:18.651624 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.651630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.651714 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.651789 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.654133 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.655267 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.657189 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.657898 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.658023 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.662530 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.662538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.662543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.662586 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.666452 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.667510 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.667516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.667520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.667731 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.668230 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.668236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.668239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.668647 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.668895 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.669509 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.669777 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:25:18.677981 12591 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [term 1 FOLLOWER]: Refusing update from remote peer 6ef60073e6a142d4aecad0a6f8af03f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:18.679772 12706 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:18.688869 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.651147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.7.43.60:38505)
0810 19:25:18.651376 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.651382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.671201 (+ 19819us) server_negotiation.cc:407] Connection header received
0810 19:25:18.671675 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.671681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.671773 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.671848 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.673865 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.674973 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.676821 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.677568 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.677674 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.681107 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.681115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.681120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.681170 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.685255 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.686193 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.686200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.686204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.686389 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.686887 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.686892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.686895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.687252 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.687524 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.688152 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.688452 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Aug 10 19:25:18 dist-test-slave-7lbf krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389917, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
W20260810 19:25:18.705560 12553 tablet.cc:2369] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:18.710752 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.643225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.7.43.62:37297)
0810 19:25:18.643451 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.643458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.650744 (+ 7286us) server_negotiation.cc:407] Connection header received
0810 19:25:18.651478 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.651484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.651578 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.651699 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.655227 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.656404 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.672192 (+ 15788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.672867 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.672996 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.696052 (+ 23056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.696061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.696065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.696112 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.702148 (+ 6036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.705411 (+ 3263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.705419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.705423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.705629 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.706305 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.706311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.706315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.706737 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.708979 (+ 2242us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.710035 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.710364 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:25:18.712605 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef60073e6a142d4aecad0a6f8af03f7. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.712939 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:18.716027 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.659929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:38505 (local address 127.0.0.1:56164)
0810 19:25:18.660176 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:18.660194 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:18.660291 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:18.689493 (+ 29202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:18.689499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:18.690050 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:18.690414 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.690432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.692025 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.692032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.693574 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:18.693596 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.694654 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.694660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:18.694884 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.695600 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:18.695626 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:18.703037 (+ 7411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:18.712821 (+ 9784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.712826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.712830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.713467 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.714033 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:18.714037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:18.714041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:18.714229 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:18.715033 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:18.715038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:18.715205 (+ 167us) client_negotiation.cc:769] Sending connection context
0810 19:25:18.715367 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 19:25:18.715662 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 19:25:18.720764 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:18.660638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.7.43.60:38505)
0810 19:25:18.689125 (+ 28487us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.689130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:18.689149 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:25:18.689198 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:18.689203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:18.689279 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:18.689402 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:18.690654 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.691845 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.693785 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:18.694470 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:18.704283 (+ 9813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:18.704508 (+ 225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:18.704519 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:18.704524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:18.704557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:18.712628 (+ 8071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.713680 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.713686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.713689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.713891 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:18.714434 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:18.714439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:18.714442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:18.714797 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:18.714967 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:18.715668 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 19:25:18.719890 (+ 4222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28370,"thread_start_us":31828,"threads_started":1}
I20260810 19:25:18.726116 12726 mvcc.cc:205] Tried to move back new op lower bound from 7317053106890252288 to 7317053106677338112. Current Snapshot: MvccSnapshot[applied={T|T < 7317053106890252288}]
I20260810 19:25:18.728582 12517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: Bootstrap starting.
I20260810 19:25:18.734136 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.734470 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:18.738267 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.738619 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:18.738619 12517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:18.739687 12716 catalog_manager.cc:1347] Generated new cluster ID: 9123d564e5874bd08fd0c855c5c487db
I20260810 19:25:18.739800 12716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:18.744346 12517 log.cc:824] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:18.779892 12517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: No bootstrap required, opened a new log
I20260810 19:25:18.780298 12716 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:18.782310 12716 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:18.805930 12716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7: Generated new TSK 0
I20260810 19:25:18.807920 12716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:18.814191 12517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.814908 12517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc2cf01bd254b7ebfe7b19a1d54a0f6, State: Initialized, Role: FOLLOWER
I20260810 19:25:18.815712 12517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } }
I20260810 19:25:18.830134 12735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.830919 12735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:18.841270 12480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6"
I20260810 19:25:18.841683 12478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:18.842028 12517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:18.845726 12478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [term 1 FOLLOWER]: Refusing update from remote peer 6ef60073e6a142d4aecad0a6f8af03f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:18.849159 12480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef60073e6a142d4aecad0a6f8af03f7 in term 1.
I20260810 19:25:18.851807 12682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:18.909174 12735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef60073e6a142d4aecad0a6f8af03f7. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.909175 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:18.909608 12735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:18.909637 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 19:25:18.945031 7340 mini_kdc.cc:348] Time spent kdestroy: real 0.118s user 0.003s sys 0.029s
I20260810 19:25:18.991376 12766 catalog_manager.cc:1259] Loaded cluster ID: 9123d564e5874bd08fd0c855c5c487db
I20260810 19:25:18.991595 12766 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: loading cluster ID for follower catalog manager: success
I20260810 19:25:18.995575 12766 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: acquiring CA information for follower catalog manager: success
I20260810 19:25:18.998631 12766 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bc2cf01bd254b7ebfe7b19a1d54a0f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 19:25:19.012594 12651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:18.989091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.7.43.62:37297)
0810 19:25:18.989350 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.989358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.007478 (+ 18120us) server_negotiation.cc:407] Connection header received
0810 19:25:19.010744 (+ 3266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.010750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.010848 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.010929 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.012272 (+ 1343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52256: BlockingRecv error: recv got EOF from 127.0.0.1:52256 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 19:25:19.017077 12523 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:18.991104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.7.43.61:44281)
0810 19:25:18.991296 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:18.991301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.014646 (+ 23345us) server_negotiation.cc:407] Connection header received
0810 19:25:19.015527 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.015533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.015638 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.015716 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.016843 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37240: BlockingRecv error: recv got EOF from 127.0.0.1:37240 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 19:25:19.019086 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.008170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.7.43.60:38505)
0810 19:25:19.008429 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.008438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.015235 (+ 6797us) server_negotiation.cc:407] Connection header received
0810 19:25:19.017779 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.017786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.017868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.017936 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.018853 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56178: BlockingRecv error: recv got EOF from 127.0.0.1:56178 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:25:19.074121 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.054365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.7.43.61:44281)
0810 19:25:19.054567 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.054574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.065761 (+ 11187us) server_negotiation.cc:407] Connection header received
0810 19:25:19.066379 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.066386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.066653 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.066813 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.067598 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.068828 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.070916 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.071600 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.071702 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.072502 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.073237 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.073360 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.073665 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.073766 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 19:25:19.125216 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.111655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.7.43.60:38505)
0810 19:25:19.111912 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.111919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.122620 (+ 10701us) server_negotiation.cc:407] Connection header received
0810 19:25:19.123361 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.123369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.123552 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.123668 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.124946 (+ 1278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56188: BlockingRecv error: recv got EOF from 127.0.0.1:56188 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 19:25:19.179602 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.161820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.7.43.61:44281)
0810 19:25:19.162048 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.162054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.172622 (+ 10568us) server_negotiation.cc:407] Connection header received
0810 19:25:19.173116 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.173123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.173228 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.173300 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.174044 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.175178 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.177200 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.177942 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.178036 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.178697 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.178858 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.178923 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.179201 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.179299 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 19:25:19.202232 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.199825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.7.43.60:38505)
0810 19:25:19.200005 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.200010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.200563 (+ 553us) server_negotiation.cc:407] Connection header received
0810 19:25:19.200813 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.200820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.200927 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.201048 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.201967 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56198: BlockingRecv error: recv got EOF from 127.0.0.1:56198 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:25:19.214979 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.207746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.7.43.61:44281)
0810 19:25:19.207937 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.207943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.208355 (+ 412us) server_negotiation.cc:407] Connection header received
0810 19:25:19.208578 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.208585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.208676 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.208784 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.209421 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.210624 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.212788 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.213417 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.213573 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.214088 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.214265 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.214356 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.214562 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.214645 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 19:25:19.226862 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.224844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.7.43.60:38505)
0810 19:25:19.225024 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.225030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.225377 (+ 347us) server_negotiation.cc:407] Connection header received
0810 19:25:19.225602 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.225607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.225714 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.225805 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.226613 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56204: BlockingRecv error: recv got EOF from 127.0.0.1:56204 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:25:19.240244 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.235165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.7.43.61:44281)
0810 19:25:19.235387 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.235394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.235693 (+ 299us) server_negotiation.cc:407] Connection header received
0810 19:25:19.235880 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.235886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.235999 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.236078 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.236470 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.237217 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.238579 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.239055 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.239173 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.239587 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.239689 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.239765 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.239898 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.239983 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260810 19:25:19.247926 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.245923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.7.43.60:38505)
0810 19:25:19.246151 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.246155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.246602 (+ 447us) server_negotiation.cc:407] Connection header received
0810 19:25:19.246727 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.246734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.246848 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.246964 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.247629 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56214: BlockingRecv error: recv got EOF from 127.0.0.1:56214 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 19:25:19.263736 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.258560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.7.43.61:44281)
0810 19:25:19.258722 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.258728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.259064 (+ 336us) server_negotiation.cc:407] Connection header received
0810 19:25:19.259249 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.259254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.259321 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.259397 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.259771 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.260540 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.261902 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.262528 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.262636 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.263018 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.263151 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.263227 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.263356 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.263432 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 19:25:19.272746 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.270667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.7.43.60:38505)
0810 19:25:19.270862 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.270868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.271224 (+ 356us) server_negotiation.cc:407] Connection header received
0810 19:25:19.271412 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.271416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.271489 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.271569 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.272494 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56218: BlockingRecv error: recv got EOF from 127.0.0.1:56218 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 19:25:19.298648 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.292184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.7.43.61:44281)
0810 19:25:19.292399 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.292406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.292646 (+ 240us) server_negotiation.cc:407] Connection header received
0810 19:25:19.292826 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.292834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.292931 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.293103 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.293785 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.294671 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.296812 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.297260 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.297354 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.297937 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.298052 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.298116 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.298310 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.298385 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 19:25:19.307377 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.305221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.7.43.60:38505)
0810 19:25:19.305425 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.305432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.306008 (+ 576us) server_negotiation.cc:407] Connection header received
0810 19:25:19.306161 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.306168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.306267 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.306389 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.307130 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56220: BlockingRecv error: recv got EOF from 127.0.0.1:56220 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:25:19.352857 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.347411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.7.43.61:44281)
0810 19:25:19.347625 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.347631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.348116 (+ 485us) server_negotiation.cc:407] Connection header received
0810 19:25:19.348314 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.348319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.348389 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.348474 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.348940 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.349777 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.351116 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.351604 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.351724 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.352125 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.352232 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.352320 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.352475 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.352564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 19:25:19.360905 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.358980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.7.43.60:38505)
0810 19:25:19.359185 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.359191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.359460 (+ 269us) server_negotiation.cc:407] Connection header received
0810 19:25:19.359662 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.359666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.359741 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.359820 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.360681 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56232: BlockingRecv error: recv got EOF from 127.0.0.1:56232 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:25:19.411191 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef60073e6a142d4aecad0a6f8af03f7" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 37297 } } peers { permanent_uuid: "8bc2cf01bd254b7ebfe7b19a1d54a0f6" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 44281 } } peers { permanent_uuid: "a8eaac23ac3d4df5a69e51c143f56c1a" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 38505 } } }
I20260810 19:25:19.411556 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef60073e6a142d4aecad0a6f8af03f7 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:19.435830 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.429002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.7.43.61:44281)
0810 19:25:19.429195 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.429202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.429830 (+ 628us) server_negotiation.cc:407] Connection header received
0810 19:25:19.430031 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.430037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.430129 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.430232 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.430750 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.431909 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.433861 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.434517 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.434656 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.435086 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.435222 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.435303 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.435431 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.435519 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 19:25:19.444980 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:19.443169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.7.43.60:38505)
0810 19:25:19.443373 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.443379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.443720 (+ 341us) server_negotiation.cc:407] Connection header received
0810 19:25:19.443873 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.443879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.443977 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.444072 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:19.444758 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56242: BlockingRecv error: recv got EOF from 127.0.0.1:56242 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:25:19.571722 12677 catalog_manager.cc:1259] Loaded cluster ID: 9123d564e5874bd08fd0c855c5c487db
I20260810 19:25:19.571822 12677 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: loading cluster ID for follower catalog manager: success
I20260810 19:25:19.574963 12677 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: acquiring CA information for follower catalog manager: success
I20260810 19:25:19.577306 12677 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8eaac23ac3d4df5a69e51c143f56c1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:25:19.581533 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.576629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.7.43.61:44281)
0810 19:25:19.576883 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.576889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.577254 (+ 365us) server_negotiation.cc:407] Connection header received
0810 19:25:19.577474 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.577480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.577582 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.577665 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.578053 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.578781 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.580108 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.580522 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.580615 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.580922 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.581018 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.581078 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.581167 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.581232 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 19:25:19.592087 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.586732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.7.43.60:38505)
0810 19:25:19.586888 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.586892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.587313 (+ 421us) server_negotiation.cc:407] Connection header received
0810 19:25:19.587465 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.587469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.587598 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.587692 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.588002 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.588659 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.590034 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.590456 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.590568 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.590986 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.591513 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.591629 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.591751 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.591814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:25:19.606637 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.598214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.7.43.61:44281)
0810 19:25:19.598440 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.598447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.600276 (+ 1829us) server_negotiation.cc:407] Connection header received
0810 19:25:19.600423 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.600429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.600521 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.600620 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.601135 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.602351 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.604406 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.605088 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.605212 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.605794 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.605930 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.606017 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.606273 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.606371 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:25:19.610294 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.599125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.7.43.60:38505)
0810 19:25:19.599324 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.599330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.604661 (+ 5331us) server_negotiation.cc:407] Connection header received
0810 19:25:19.604814 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.604819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.604906 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.604992 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.605871 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.606829 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.608352 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.609058 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.609177 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.609595 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.609724 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.609792 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.609955 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.610030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:25:19.611922 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:19.598104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.7.43.62:37297)
0810 19:25:19.605300 (+ 7196us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:19.605307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:19.605326 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:25:19.605451 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:19.605457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:19.605738 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:19.605927 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:19.606873 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.607881 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.609694 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:19.610288 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:19.610391 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:19.610849 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:19.611275 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:19.611386 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:19.611545 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 19:25:19.611632 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7071,"thread_start_us":6961,"threads_started":1}
I20260810 19:25:19.616940 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12288
I20260810 19:25:19.617882 12388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:19.726681 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12288
I20260810 19:25:19.754540 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12409
I20260810 19:25:19.755631 12508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:19.861990 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12409
I20260810 19:25:19.889566 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12526
I20260810 19:25:19.890811 12629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:19.996037 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12526
2026-08-10T19:25:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4634 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:25:20 dist-test-slave-7lbf krb5kdc[12965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:25:20 dist-test-slave-7lbf krb5kdc[12965](info): set up 2 sockets
Aug 10 19:25:20 dist-test-slave-7lbf krb5kdc[12965](info): commencing operation
krb5kdc: starting...
W20260810 19:25:21.263818 7340 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/test-user.keytab.
Aug 10 19:25:21 dist-test-slave-7lbf krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.43.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
Entry for principal kudu/127.7.43.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab.
I20260810 19:25:21.592633 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39893
--webserver_interface=127.7.43.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39893,127.7.43.61:43381,127.7.43.60:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:25:21.936944 12986 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:25:21.937300 12986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:21.937402 12986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:21.937557 12986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:21.944873 12986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:21.944975 12986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:21.945072 12986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:21.945139 12986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:21.945225 12986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:21.955235 12986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39893,127.7.43.61:43381,127.7.43.60:35287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.62.keytab
--principal=kudu/127.7.43.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.62:39893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.43.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:21.956637 12986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:21.958065 12986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:21.969625 12996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:21.969691 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:21.969882 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:21.970628 12986 server_base.cc:1034] running on GCE node
I20260810 19:25:21.971470 12986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:21.972914 12986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:21.974134 12986 hybrid_clock.cc:648] HybridClock initialized: now 1786389921974064 us; error 37 us; skew 500 ppm
I20260810 19:25:21.974474 12986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:21 dist-test-slave-7lbf krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:21.980583 12986 init.cc:377] Logged in from keytab as kudu/127.7.43.62@KRBTEST.COM (short username kudu)
I20260810 19:25:21.982177 12986 webserver.cc:460] Webserver started at http://127.7.43.62:43265/ using document root <none> and password file <none>
I20260810 19:25:21.982890 12986 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:21.982998 12986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:21.983305 12986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:21.986384 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data/instance:
uuid: "1c5fc5bdeec14c20b93b6788546fc8c3"
format_stamp: "Formatted at 2026-08-10 19:25:21 on dist-test-slave-7lbf"
server_key: "7052fbfb7d3c393569b00ea5e3807d8f"
server_key_iv: "f4f8c4478b10e63924485131262e71b1"
server_key_version: "encryptionkey@0"
I20260810 19:25:21.987157 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal/instance:
uuid: "1c5fc5bdeec14c20b93b6788546fc8c3"
format_stamp: "Formatted at 2026-08-10 19:25:21 on dist-test-slave-7lbf"
server_key: "7052fbfb7d3c393569b00ea5e3807d8f"
server_key_iv: "f4f8c4478b10e63924485131262e71b1"
server_key_version: "encryptionkey@0"
I20260810 19:25:21.992343 12986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:21.995790 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:21.997300 12986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 19:25:21.997507 12986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal
uuid: "1c5fc5bdeec14c20b93b6788546fc8c3"
format_stamp: "Formatted at 2026-08-10 19:25:21 on dist-test-slave-7lbf"
server_key: "7052fbfb7d3c393569b00ea5e3807d8f"
server_key_iv: "f4f8c4478b10e63924485131262e71b1"
server_key_version: "encryptionkey@0"
I20260810 19:25:21.997849 12986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:22.020272 12986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:22.024343 12986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:22.024612 12986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:22.044253 12986 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.62:39893
I20260810 19:25:22.044276 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.62:39893 every 8 connection(s)
I20260810 19:25:22.045606 12986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/data/info.pb
I20260810 19:25:22.049156 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:22.054387 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 12986
I20260810 19:25:22.054587 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-0/wal/instance
I20260810 19:25:22.055164 7340 external_mini_cluster.cc:1428] Setting key 5a78d1d15716131f439a248fc9aa57a5
I20260810 19:25:22.056243 13093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:22 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:22.078310 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.059836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.7.43.62:39893)
0810 19:25:22.061481 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:22.061488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:22.061565 (+ 77us) server_negotiation.cc:407] Connection header received
0810 19:25:22.063084 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:22.063090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:22.063331 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:22.063501 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:22.064780 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.066026 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.067342 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.067875 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.068063 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.071498 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:22.071507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:22.071520 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:22.071551 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:22.074193 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.074959 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.074965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.074968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.075123 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.075566 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.075569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.075571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.076062 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:22.076213 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:22.076820 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 19:25:22.077150 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1445,"thread_start_us":1228,"threads_started":1}
I20260810 19:25:22.078351 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.059835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39893 (local address 127.0.0.1:41460)
0810 19:25:22.060724 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.060769 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:22.062720 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:22.063555 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:22.063562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:22.064237 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:22.064586 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.064604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.066170 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.066174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.067210 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.067223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.068097 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.068104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.068274 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.068880 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:22.068901 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:22.071245 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:22.074325 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.074337 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.074348 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.074847 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.075227 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.075231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.075234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.075456 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.076196 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:22.076203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:22.076440 (+ 237us) client_negotiation.cc:769] Sending connection context
0810 19:25:22.076741 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 19:25:22.077310 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":105,"thread_start_us":374,"threads_started":1}
I20260810 19:25:22.081343 13093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:22.082481 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.082018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37354)
0810 19:25:22.082206 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.082361 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 19:25:22.082765 13012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
I20260810 19:25:22.084287 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.083895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37362)
0810 19:25:22.084048 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.084139 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:25:22.084501 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:43381: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.7.43.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab.
Entry for principal kudu/127.7.43.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab.
I20260810 19:25:22.128363 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:43381
--webserver_interface=127.7.43.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab
--principal=kudu/127.7.43.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39893,127.7.43.61:43381,127.7.43.60:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:25:22.133850 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } attempt: 1
I20260810 19:25:22.135114 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.134605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37366)
0810 19:25:22.134816 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.134969 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 19:25:22.136092 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.135731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37380)
0810 19:25:22.135889 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.135959 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:25:22.136269 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:43381: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
I20260810 19:25:22.204536 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } attempt: 2
I20260810 19:25:22.205962 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.205293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37384)
0810 19:25:22.205609 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.205750 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 19:25:22.207029 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.206607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37394)
0810 19:25:22.206750 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.206852 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:25:22.207234 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:43381: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
I20260810 19:25:22.298542 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } attempt: 3
I20260810 19:25:22.300010 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.299422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37400)
0810 19:25:22.299676 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.299806 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 19:25:22.301137 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.300721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37408)
0810 19:25:22.300877 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.300966 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 19:25:22.301328 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:43381: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
I20260810 19:25:22.444614 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } attempt: 4
I20260810 19:25:22.446136 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.445450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37412)
0810 19:25:22.445889 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.445990 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260810 19:25:22.447202 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.446825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37428)
0810 19:25:22.446986 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.447064 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 19:25:22.447439 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.61:43381: Network error: Client connection negotiation failed: client connection to 127.7.43.61:43381: connect: Connection refused (error 111)
W20260810 19:25:22.471182 13105 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:25:22.471539 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:22.471635 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:22.471751 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:22.479194 13105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:22.479296 13105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:22.479382 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:22.479449 13105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:22.479534 13105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:22.489451 13105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39893,127.7.43.61:43381,127.7.43.60:35287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.61.keytab
--principal=kudu/127.7.43.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.61:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.43.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:22.490875 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:22.492242 13105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:22.504071 13115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:22.504107 13118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:22.504045 13116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:22.505314 13105 server_base.cc:1034] running on GCE node
I20260810 19:25:22.506357 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:22.507875 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:22.509105 13105 hybrid_clock.cc:648] HybridClock initialized: now 1786389922509036 us; error 47 us; skew 500 ppm
I20260810 19:25:22.509444 13105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:22 dist-test-slave-7lbf krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389922, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:22.512598 13105 init.cc:377] Logged in from keytab as kudu/127.7.43.61@KRBTEST.COM (short username kudu)
I20260810 19:25:22.514165 13105 webserver.cc:460] Webserver started at http://127.7.43.61:40109/ using document root <none> and password file <none>
I20260810 19:25:22.514868 13105 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:22.514972 13105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:22.515282 13105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:22.518353 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data/instance:
uuid: "5e792fad2ceb4eb09f6098e4d9996262"
format_stamp: "Formatted at 2026-08-10 19:25:22 on dist-test-slave-7lbf"
server_key: "1dc198d58a605b451d02ed6d95023397"
server_key_iv: "1ccf18de4a7f492710f1c5e83948129c"
server_key_version: "encryptionkey@0"
I20260810 19:25:22.519140 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal/instance:
uuid: "5e792fad2ceb4eb09f6098e4d9996262"
format_stamp: "Formatted at 2026-08-10 19:25:22 on dist-test-slave-7lbf"
server_key: "1dc198d58a605b451d02ed6d95023397"
server_key_iv: "1ccf18de4a7f492710f1c5e83948129c"
server_key_version: "encryptionkey@0"
I20260810 19:25:22.524451 13105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 19:25:22.527951 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:22.529214 13105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 19:25:22.529389 13105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal
uuid: "5e792fad2ceb4eb09f6098e4d9996262"
format_stamp: "Formatted at 2026-08-10 19:25:22 on dist-test-slave-7lbf"
server_key: "1dc198d58a605b451d02ed6d95023397"
server_key_iv: "1ccf18de4a7f492710f1c5e83948129c"
server_key_version: "encryptionkey@0"
I20260810 19:25:22.529633 13105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:22.566403 13105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:22.570868 13105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:22.571163 13105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:22.590240 13105 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.61:43381
I20260810 19:25:22.590265 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.61:43381 every 8 connection(s)
I20260810 19:25:22.591738 13105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/data/info.pb
I20260810 19:25:22.595536 13218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:22.601910 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13105
I20260810 19:25:22.602110 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-1/wal/instance
I20260810 19:25:22.602711 7340 external_mini_cluster.cc:1428] Setting key 37ebb2ffa04a716f3728c747bf2819bd
I20260810 19:25:22.602411 13218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:22 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389922, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:22.625473 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.605676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.7.43.62:39893)
0810 19:25:22.606237 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:22.606243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:22.607229 (+ 986us) server_negotiation.cc:407] Connection header received
0810 19:25:22.609544 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:22.609551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:22.609627 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:22.609725 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:22.611750 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.612515 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.614595 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.615039 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.615135 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.619161 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:22.619168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:22.619173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:22.619221 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:22.622397 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.623137 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.623143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.623146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.623304 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.623713 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.623717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.623720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.624076 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:22.624250 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:22.624910 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 19:25:22.625127 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":284,"threads_started":1}
I20260810 19:25:22.626384 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.606193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39893 (local address 127.0.0.1:41476)
0810 19:25:22.607038 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.607094 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:22.609232 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:22.609987 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:22.610008 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:22.610750 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:22.611518 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.611544 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.612680 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.612685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.614435 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.614455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.615265 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.615272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.615552 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.616257 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:22.616286 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:22.618902 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:22.622533 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.622541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.622554 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.622997 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.623407 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.623410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.623412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.623600 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.624190 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:22.624195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:22.624532 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 19:25:22.624764 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 19:25:22.625187 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":323,"threads_started":1}
I20260810 19:25:22.628118 13218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:22 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389922, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.43.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.43.60@KRBTEST.COM" created.
I20260810 19:25:22.642725 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.628849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37436)
0810 19:25:22.629009 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.629028 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:22.629126 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:22.630193 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:22.630197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:22.630621 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:22.630749 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.630758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.632106 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.632111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.632969 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.632981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.633909 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.633915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.634135 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.634749 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:22.634765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:22.637103 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:22.640175 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.640179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.640182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.640612 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.640972 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.640977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.640981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.641130 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.641774 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:22.641778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:22.641908 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 19:25:22.642031 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 19:25:22.642303 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260810 19:25:22.642795 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.629108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.7.43.61:43381)
0810 19:25:22.629800 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:22.629806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:22.629831 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:25:22.629872 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:22.629877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:22.630052 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:22.630182 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:22.630906 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.631961 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.633113 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.633771 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.633900 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.637310 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:22.637322 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:22.637336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:22.637364 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:22.640059 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.640730 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.640737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.640739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.640856 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.641228 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.641233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.641236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.641642 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:22.641824 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:22.642183 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 19:25:22.642397 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":536,"thread_start_us":407,"threads_started":1}
I20260810 19:25:22.645135 13218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:22.646394 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.645856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36622)
0810 19:25:22.646035 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.646193 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 19:25:22.646677 13135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.648216 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.647858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36636)
0810 19:25:22.648010 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.648098 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:25:22.648453 13218 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.43.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab.
Entry for principal kudu/127.7.43.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab.
I20260810 19:25:22.678948 7340 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
/tmp/dist-test-taskuOTVHW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:35287
--webserver_interface=127.7.43.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab
--principal=kudu/127.7.43.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.43.20:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.43.62:39893,127.7.43.61:43381,127.7.43.60:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:25:22.697789 13218 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 1
I20260810 19:25:22.698964 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.698516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36646)
0810 19:25:22.698718 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.698804 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 19:25:22.700045 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.699700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36662)
0810 19:25:22.699861 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.699939 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 19:25:22.700261 13218 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.746730 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } attempt: 5
Aug 10 19:25:22 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:25:22.761667 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.747672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.7.43.61:43381)
0810 19:25:22.747839 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:22.747844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:22.747871 (+ 27us) server_negotiation.cc:407] Connection header received
0810 19:25:22.747942 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:22.747946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:22.748007 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:22.748090 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:22.749097 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.749835 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.751570 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.752036 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.752134 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.756046 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:22.756051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:22.756054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:22.756089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:22.758775 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.759584 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.759589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.759591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.759728 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:22.760156 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:22.760160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:22.760162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:22.760599 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:22.760733 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:22.761192 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 19:25:22.761379 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:25:22.761798 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.747418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37442)
0810 19:25:22.747662 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.747685 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:22.747825 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:22.748162 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:22.748167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:22.748713 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:22.748957 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.748971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.749992 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.749998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.751390 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:22.751408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:22.752194 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:22.752200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:22.752397 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:22.753135 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:22.753159 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:22.755833 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:22.758924 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.758929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.758933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.759436 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.759840 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:22.759845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:22.759849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:22.760034 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:22.760733 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:22.760737 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:22.760927 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 19:25:22.761084 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 19:25:22.761408 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 19:25:22.762907 13093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:22.764019 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.763534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36678)
0810 19:25:22.763728 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.763835 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:25:22.765007 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.764636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36686)
0810 19:25:22.764776 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.764875 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 19:25:22.765249 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.768465 13218 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 2
I20260810 19:25:22.769382 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.768988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36690)
0810 19:25:22.769155 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.769247 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 19:25:22.770289 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.769959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36692)
0810 19:25:22.770108 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.770176 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 19:25:22.770488 13218 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.816488 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 1
I20260810 19:25:22.817853 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.817280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36698)
0810 19:25:22.817496 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.817649 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 19:25:22.818912 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.818557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36708)
0810 19:25:22.818713 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.818785 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 19:25:22.819123 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.861719 13218 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 3
I20260810 19:25:22.863055 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.862539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36720)
0810 19:25:22.862748 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.862884 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 19:25:22.864068 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.863655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36724)
0810 19:25:22.863878 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.863954 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260810 19:25:22.864276 13218 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.887353 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 2
I20260810 19:25:22.888525 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.888032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36732)
0810 19:25:22.888266 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.888363 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 19:25:22.889586 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.889213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36734)
0810 19:25:22.889358 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.889424 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:25:22.889787 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:22.996035 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 3
I20260810 19:25:22.997191 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.996714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36746)
0810 19:25:22.996923 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.997029 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 19:25:22.998474 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:22.997955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36748)
0810 19:25:22.998261 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:22.998351 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260810 19:25:22.998706 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
I20260810 19:25:23.007486 13218 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 4
I20260810 19:25:23.008576 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.008088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36762)
0810 19:25:23.008326 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.008414 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 19:25:23.009476 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.009164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36764)
0810 19:25:23.009301 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.009368 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:25:23.009717 13218 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.43.60:35287: Network error: Client connection negotiation failed: client connection to 127.7.43.60:35287: connect: Connection refused (error 111)
W20260810 19:25:23.031178 13226 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:25:23.031533 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:25:23.031631 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:25:23.031744 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:25:23.039098 13226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:25:23.039203 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:25:23.039294 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:25:23.039359 13226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:25:23.039443 13226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:25:23.049439 13226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.43.20:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.43.62:39893,127.7.43.61:43381,127.7.43.60:35287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/krb5kdc/kudu_127.7.43.60.keytab
--principal=kudu/127.7.43.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.43.60:35287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.43.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 18:32:41 UTC on ccycloud.quasar-qrrker.root.comops.site
build id 4578684
ASAN enabled
I20260810 19:25:23.050719 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:25:23.052004 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:25:23.063066 13232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:23.063114 13237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:25:23.063136 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:25:23.063987 13226 server_base.cc:1034] running on GCE node
I20260810 19:25:23.064708 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:25:23.066119 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:25:23.067329 13226 hybrid_clock.cc:648] HybridClock initialized: now 1786389923067264 us; error 43 us; skew 500 ppm
I20260810 19:25:23.067639 13226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389923, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:25:23.070725 13226 init.cc:377] Logged in from keytab as kudu/127.7.43.60@KRBTEST.COM (short username kudu)
I20260810 19:25:23.072141 13226 webserver.cc:460] Webserver started at http://127.7.43.60:45611/ using document root <none> and password file <none>
I20260810 19:25:23.072845 13226 fs_manager.cc:359] Metadata directory not provided
I20260810 19:25:23.072957 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:25:23.073267 13226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:25:23.076216 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data/instance:
uuid: "4f7903f02abb445292ada5a8cbc08fd0"
format_stamp: "Formatted at 2026-08-10 19:25:23 on dist-test-slave-7lbf"
server_key: "d2a6748afb805792cc2f4d1d81c48935"
server_key_iv: "9679bb5d03af914cdcdb43a745045f83"
server_key_version: "encryptionkey@0"
I20260810 19:25:23.076980 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal/instance:
uuid: "4f7903f02abb445292ada5a8cbc08fd0"
format_stamp: "Formatted at 2026-08-10 19:25:23 on dist-test-slave-7lbf"
server_key: "d2a6748afb805792cc2f4d1d81c48935"
server_key_iv: "9679bb5d03af914cdcdb43a745045f83"
server_key_version: "encryptionkey@0"
I20260810 19:25:23.082015 13226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 19:25:23.085363 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:25:23.086692 13226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:25:23.086898 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data,/tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal
uuid: "4f7903f02abb445292ada5a8cbc08fd0"
format_stamp: "Formatted at 2026-08-10 19:25:23 on dist-test-slave-7lbf"
server_key: "d2a6748afb805792cc2f4d1d81c48935"
server_key_iv: "9679bb5d03af914cdcdb43a745045f83"
server_key_version: "encryptionkey@0"
I20260810 19:25:23.087098 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:25:23.115947 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:25:23.120251 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:25:23.120508 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:25:23.137610 13226 rpc_server.cc:307] RPC server started. Bound to: 127.7.43.60:35287
I20260810 19:25:23.137621 13325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.43.60:35287 every 8 connection(s)
I20260810 19:25:23.139003 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/data/info.pb
I20260810 19:25:23.142820 13326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:25:23.144188 7340 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu as pid 13226
I20260810 19:25:23.144414 7340 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuOTVHW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786389541863099-7340-0/minicluster-data/master-2/wal/instance
I20260810 19:25:23.144871 7340 external_mini_cluster.cc:1428] Setting key f88c5ea0d1aa7db8e6056737abeea31f
I20260810 19:25:23.152657 13326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389923, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
I20260810 19:25:23.168705 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.146750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.7.43.60:35287)
0810 19:25:23.147567 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.147583 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.147641 (+ 58us) server_negotiation.cc:407] Connection header received
0810 19:25:23.149380 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.149413 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.149981 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.150472 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.151832 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.153859 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.155205 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.156146 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.156909 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.160963 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.160972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.160985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.161025 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.164350 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.165115 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.165122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.165129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.165269 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.165694 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.165697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.165699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.166359 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.166592 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.167094 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.167488 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":340,"threads_started":1}
I20260810 19:25:23.171000 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.155664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.7.43.62:39893)
0810 19:25:23.156284 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.156290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.156813 (+ 523us) server_negotiation.cc:407] Connection header received
0810 19:25:23.157027 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.157031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.157123 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.157230 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.158474 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.159439 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.160903 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.161315 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.161406 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.165075 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.165081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.165086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.165128 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.168044 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.168939 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.168944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.168947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.169100 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.169477 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.169482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.169485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.169785 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.169920 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.170377 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.170600 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":308,"threads_started":1}
I20260810 19:25:23.171012 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.155693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.62:39893 (local address 127.0.0.1:41490)
0810 19:25:23.156649 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.156708 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:23.156882 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:23.157319 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:23.157324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:23.158050 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:23.158304 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.158314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.159600 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.159606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.160738 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.160754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.161501 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.161508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.161691 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.162258 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:23.162280 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:23.164855 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:23.168166 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.168176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.168183 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.168725 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.169215 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.169221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.169225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.169373 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.169922 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:23.169931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:23.170089 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 19:25:23.170251 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 19:25:23.170602 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":528,"threads_started":1}
I20260810 19:25:23.172689 13326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:23.175933 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 4
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389923, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.62@KRBTEST.COM
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.62@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
I20260810 19:25:23.187498 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.173591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.7.43.61:43381)
0810 19:25:23.173763 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.173768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.173784 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:25:23.173895 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.173900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.173979 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.174087 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.174702 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.175426 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.177033 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.177659 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.177752 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.181854 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.181860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.181865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.181906 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.184601 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.185230 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.185235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.185239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.185373 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.185843 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.185846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.185848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.186124 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.186267 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.187007 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.187163 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:25:23.188542 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.173475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.61:43381 (local address 127.0.0.1:37456)
0810 19:25:23.173671 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.173690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:23.173781 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:23.174101 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:23.174105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:23.174416 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:23.174562 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.174570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.175581 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.175586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.176830 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.176844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.177809 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.177814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.178012 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.178614 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:23.178638 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:23.181630 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:23.184721 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.184726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.184731 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.185110 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.185483 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.185488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.185491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.185706 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.186330 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:23.186335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:23.186496 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 19:25:23.186627 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 19:25:23.188197 (+ 1570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 19:25:23.188692 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.171894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.7.43.62:39893)
0810 19:25:23.172159 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.172164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.172195 (+ 31us) server_negotiation.cc:407] Connection header received
0810 19:25:23.172249 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.172253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.172335 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.172429 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.173940 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.174735 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.176060 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.176649 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.176771 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.182045 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.182052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.182057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.182091 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.185379 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.186203 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.186208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.186211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.186364 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.186777 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.186782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.186785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.187170 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.187450 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.188262 (+ 812us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.188446 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260810 19:25:23.189668 13326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:25:23.191942 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.176515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36788)
0810 19:25:23.176749 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.176767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:23.176854 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:23.177363 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:23.177368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:23.177817 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:23.178010 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.178022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.179071 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.179077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.180173 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.180188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.180969 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.180976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.181161 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.181783 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:23.181805 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:23.184644 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:23.188175 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.188179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.188183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.189672 (+ 1489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.190200 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.190203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.190206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.190376 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.191035 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:23.191040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:23.191176 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 19:25:23.191318 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 19:25:23.191601 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260810 19:25:23.192139 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.176568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.7.43.60:35287)
0810 19:25:23.177051 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.177056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.177072 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:25:23.177113 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.177117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.177180 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.177313 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.178155 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.178937 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.180330 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.180811 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.180908 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.184878 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.184885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.184890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.184928 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.188049 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.189822 (+ 1773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.189827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.189830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.190062 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.190486 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.190499 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.190502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.190898 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.191034 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.191463 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.191711 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378}
I20260810 19:25:23.198591 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3: Bootstrap starting.
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389923, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.60@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.43.61@KRBTEST.COM
I20260810 19:25:23.202607 13093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:23.203943 13093 log.cc:824] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:23.206269 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.190670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36796)
0810 19:25:23.190977 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.190994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:23.191083 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:23.192217 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:23.192222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:23.192699 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:23.192863 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.192875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.194253 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.194259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.195271 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.195285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.196301 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.196306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.196482 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.197034 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:23.197055 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:23.199625 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:23.203446 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.203451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.203455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.203967 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.204380 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.204385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.204388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.204562 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.205214 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:23.205219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:23.205377 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 19:25:23.205568 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 19:25:23.205890 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260810 19:25:23.206506 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.191014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.7.43.60:35287)
0810 19:25:23.191710 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.191715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.191739 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:25:23.191783 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.191788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.191856 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.191954 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.193050 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.194138 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.195417 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.196009 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.196124 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.199849 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.199856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.199861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.199897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.203300 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.204099 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.204104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.204108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.204256 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.204697 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.204701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.204704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.205068 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.205205 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.205684 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.206067 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":568,"thread_start_us":319,"threads_started":1}
I20260810 19:25:23.207394 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3: No bootstrap required, opened a new log
I20260810 19:25:23.207998 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.190693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.7.43.61:43381)
0810 19:25:23.190854 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.190859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.191822 (+ 963us) server_negotiation.cc:407] Connection header received
0810 19:25:23.191877 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.191882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.191954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.192038 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.193714 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.194703 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.196085 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.196521 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.196630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.201087 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.201094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.201099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.201136 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.204278 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.205148 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.205154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.205157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.205380 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.205838 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.205843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.205847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.206206 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.206483 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.207528 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.207768 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:25:23.212621 7340 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:25:23.212668 13326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: Bootstrap starting.
I20260810 19:25:23.215025 13057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41498
I20260810 19:25:23.216712 13326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:23.218103 13326 log.cc:824] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:23.218674 13093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.219074 13093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5fc5bdeec14c20b93b6788546fc8c3, State: Initialized, Role: FOLLOWER
I20260810 19:25:23.219609 13093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.220592 13343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.221016 13343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.221727 13326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: No bootstrap required, opened a new log
I20260810 19:25:23.221962 13093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:23.222041 13057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 19:25:23.222241 13057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:23.224881 13057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.226311 13057 leader_election.cc:288] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [CANDIDATE]: Term 1 election: Requested vote from peers 5e792fad2ceb4eb09f6098e4d9996262 (127.7.43.61:43381), 4f7903f02abb445292ada5a8cbc08fd0 (127.7.43.60:35287)
I20260810 19:25:23.233264 13326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.233935 13326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7903f02abb445292ada5a8cbc08fd0, State: Initialized, Role: FOLLOWER
I20260810 19:25:23.234506 13326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.238461 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.238972 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.239856 13326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:23.240342 13286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7903f02abb445292ada5a8cbc08fd0"
I20260810 19:25:23.240893 13286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:23.244491 13286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c5fc5bdeec14c20b93b6788546fc8c3 in term 1.
I20260810 19:25:23.245371 13016 leader_election.cc:302] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5fc5bdeec14c20b93b6788546fc8c3, 4f7903f02abb445292ada5a8cbc08fd0; no voters:
I20260810 19:25:23.245936 13343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:25:23.266340 13343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1c5fc5bdeec14c20b93b6788546fc8c3, State: Running, Role: LEADER
I20260810 19:25:23.267189 13343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.269918 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.230811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.7.43.62:39893)
0810 19:25:23.230992 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.230999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.232156 (+ 1157us) server_negotiation.cc:407] Connection header received
0810 19:25:23.232748 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.232754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.232844 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.232920 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.238868 (+ 5948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.240086 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.252726 (+ 12640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.253470 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.253620 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.259244 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.259251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.259256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.259334 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.264430 (+ 5096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.265477 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.265503 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.265510 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.265882 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.266483 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.266487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.266490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.267143 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.268619 (+ 1476us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.268857 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.269210 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:25:23.276042 13371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c5fc5bdeec14c20b93b6788546fc8c3. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.276607 13371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: This master's current role is: LEADER
W20260810 19:25:23.279316 13019 tablet.cc:2369] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:25:23.280617 13395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:25:23.284291 13393 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:25:23.286370 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.231612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.7.43.61:43381)
0810 19:25:23.231785 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.231790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.248388 (+ 16598us) server_negotiation.cc:407] Connection header received
0810 19:25:23.249160 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.249168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.249267 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.249356 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.253800 (+ 4444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.255205 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.258296 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.258978 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.259075 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.279237 (+ 20162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.279246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.279250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.279319 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.282893 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.283736 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.283743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.283747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.283948 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.284514 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.284520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.284523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.284913 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.285158 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.285750 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.286000 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:25:23.296968 13395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:25:23.289623 12986 master_runner.cc:186] Error getting master registration for 127.7.43.61:43381: OK, instance_id { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" instance_seqno: 1786389922584754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:25:23.303194 12986 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:25:23.303714 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.238282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.7.43.60:35287)
0810 19:25:23.238538 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.238547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.262422 (+ 23875us) server_negotiation.cc:407] Connection header received
0810 19:25:23.262512 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.262518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.262614 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.262712 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.276437 (+ 13725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.277633 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.283664 (+ 6031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.284290 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.284403 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.287584 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.287591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.287595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.287635 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.299448 (+ 11813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.300817 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.300824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.300828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.300988 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.301611 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.301616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.301619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.302138 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.302458 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.303079 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.303333 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 19:25:23.308902 13218 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } attempt: 5
I20260810 19:25:23.304757 13226 master_runner.cc:186] Error getting master registration for 127.7.43.61:43381: OK, instance_id { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" instance_seqno: 1786389922584754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:25:23.315766 13226 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 19:25:23.318341 13402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:25:23.318511 13402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 19:25:23 dist-test-slave-7lbf krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786389922, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.43.61@KRBTEST.COM for kudu/127.7.43.60@KRBTEST.COM
I20260810 19:25:23.325942 13286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [term 1 FOLLOWER]: Refusing update from remote peer 1c5fc5bdeec14c20b93b6788546fc8c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:23.328481 13371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:23.335814 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.309905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.43.60:35287 (local address 127.0.0.1:36804)
0810 19:25:23.310176 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 19:25:23.310197 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 19:25:23.310309 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:25:23.311562 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:25:23.311568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:25:23.312122 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:25:23.312348 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.312362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.315758 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.315764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.317152 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:25:23.317170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.320339 (+ 3169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.320346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:25:23.320568 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.321236 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:25:23.321259 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:25:23.324113 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:25:23.328454 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.328459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.328463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.329027 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.329428 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:25:23.329434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:25:23.329438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:25:23.329654 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:25:23.334691 (+ 5037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:25:23.334697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:25:23.334912 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 19:25:23.335067 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 19:25:23.335430 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 19:25:23.336794 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:23.310373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.7.43.60:35287)
0810 19:25:23.310567 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.310573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.310589 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:25:23.310646 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.310650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.310728 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.310831 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.313983 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.315535 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.319443 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:23.320159 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:23.320283 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:23.324366 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:25:23.324372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:25:23.324377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:25:23.324417 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:25:23.328291 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.329143 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.329149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.329153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.329317 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:25:23.329763 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:25:23.329769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:25:23.329773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:25:23.330105 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:25:23.330263 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:23.335375 (+ 5112us) server_negotiation.cc:299] Negotiation successful
0810 19:25:23.336011 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 19:25:23.343384 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: Bootstrap starting.
I20260810 19:25:23.349772 13218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: Neither blocks nor log segments found. Creating new log.
I20260810 19:25:23.351315 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c5fc5bdeec14c20b93b6788546fc8c3. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.351667 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.352278 13218 log.cc:824] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: Log is configured to *not* fsync() on all Append() calls
I20260810 19:25:23.355527 13405 mvcc.cc:205] Tried to move back new op lower bound from 7317053125932093440 to 7317053125709926400. Current Snapshot: MvccSnapshot[applied={T|T < 7317053125932093440}]
I20260810 19:25:23.355850 13407 mvcc.cc:205] Tried to move back new op lower bound from 7317053125932093440 to 7317053125709926400. Current Snapshot: MvccSnapshot[applied={T|T < 7317053125932093440}]
I20260810 19:25:23.368152 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.368438 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.372769 13343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.373119 13343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: This master's current role is: LEADER
I20260810 19:25:23.378331 13395 catalog_manager.cc:1347] Generated new cluster ID: e33523c02d1b4f30b34484a3faaa26fb
I20260810 19:25:23.378463 13395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:25:23.396662 13218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: No bootstrap required, opened a new log
I20260810 19:25:23.398115 13395 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:25:23.401494 13395 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:25:23.408566 13218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.408964 13218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e792fad2ceb4eb09f6098e4d9996262, State: Initialized, Role: FOLLOWER
I20260810 19:25:23.409508 13218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } }
I20260810 19:25:23.423611 13395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3: Generated new TSK 0
I20260810 19:25:23.423727 13418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.424331 13218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:25:23.424358 13175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e792fad2ceb4eb09f6098e4d9996262"
I20260810 19:25:23.424422 13418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.424747 13172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:25:23.428855 13172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [term 1 FOLLOWER]: Refusing update from remote peer 1c5fc5bdeec14c20b93b6788546fc8c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:25:23.430023 13343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:25:23.430365 13395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:25:23.434008 13175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c5fc5bdeec14c20b93b6788546fc8c3 in term 1.
I20260810 19:25:23.492556 13418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c5fc5bdeec14c20b93b6788546fc8c3. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.492928 13418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.493292 13433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.493716 13433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:25:23.505383 13442 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260810 19:25:23.541726 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.526759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.7.43.61:43381)
0810 19:25:23.526950 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.526958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.534757 (+ 7799us) server_negotiation.cc:407] Connection header received
0810 19:25:23.539893 (+ 5136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.539900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.540004 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.540081 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.541451 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37476: BlockingRecv error: recv got EOF from 127.0.0.1:37476 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:25:23.544503 13333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.526929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.7.43.62:39893)
0810 19:25:23.527192 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.527198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.542740 (+ 15542us) server_negotiation.cc:407] Connection header received
0810 19:25:23.542895 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.542910 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.543007 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.543119 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.544219 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41510: BlockingRecv error: recv got EOF from 127.0.0.1:41510 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 19:25:23.545387 13337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.532973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.7.43.60:35287)
0810 19:25:23.533186 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.533192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.543377 (+ 10185us) server_negotiation.cc:407] Connection header received
0810 19:25:23.543634 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.543640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.543725 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.543795 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.545143 (+ 1348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36818: BlockingRecv error: recv got EOF from 127.0.0.1:36818 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 19:25:23.555846 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.553804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.7.43.61:43381)
0810 19:25:23.554034 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.554042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.554482 (+ 440us) server_negotiation.cc:407] Connection header received
0810 19:25:23.554717 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.554721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.554819 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.554911 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.555599 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37492: BlockingRecv error: recv got EOF from 127.0.0.1:37492 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 19:25:23.565078 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.563076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.7.43.61:43381)
0810 19:25:23.563264 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.563271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.563748 (+ 477us) server_negotiation.cc:407] Connection header received
0810 19:25:23.563932 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.563937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.564008 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.564089 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.564847 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37504: BlockingRecv error: recv got EOF from 127.0.0.1:37504 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 19:25:23.571707 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.569811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.7.43.61:43381)
0810 19:25:23.569981 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.569987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.570433 (+ 446us) server_negotiation.cc:407] Connection header received
0810 19:25:23.570625 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.570631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.570715 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.570800 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.571466 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37518: BlockingRecv error: recv got EOF from 127.0.0.1:37518 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 19:25:23.581802 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.578504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.7.43.61:43381)
0810 19:25:23.578700 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.578706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.579198 (+ 492us) server_negotiation.cc:407] Connection header received
0810 19:25:23.579384 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.579389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.579459 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.579536 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.581636 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37524: BlockingRecv error: recv got EOF from 127.0.0.1:37524 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:25:23.627826 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.625661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.7.43.61:43381)
0810 19:25:23.625871 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.625879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.626465 (+ 586us) server_negotiation.cc:407] Connection header received
0810 19:25:23.626663 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.626669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.626745 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.626829 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.627604 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37536: BlockingRecv error: recv got EOF from 127.0.0.1:37536 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 19:25:23.649500 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.647324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.7.43.61:43381)
0810 19:25:23.647526 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.647533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.648064 (+ 531us) server_negotiation.cc:407] Connection header received
0810 19:25:23.648259 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.648263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.648353 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.648451 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.649236 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37544: BlockingRecv error: recv got EOF from 127.0.0.1:37544 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 19:25:23.693841 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.691588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.7.43.61:43381)
0810 19:25:23.691817 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.691824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.692338 (+ 514us) server_negotiation.cc:407] Connection header received
0810 19:25:23.692535 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.692540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.692616 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.692720 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.693579 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37556: BlockingRecv error: recv got EOF from 127.0.0.1:37556 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 19:25:23.769985 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.767044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.7.43.61:43381)
0810 19:25:23.767284 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.767290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.768312 (+ 1022us) server_negotiation.cc:407] Connection header received
0810 19:25:23.768525 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.768531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.768633 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.768754 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.769746 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37560: BlockingRecv error: recv got EOF from 127.0.0.1:37560 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 19:25:23.848567 13343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5fc5bdeec14c20b93b6788546fc8c3" member_type: VOTER last_known_addr { host: "127.7.43.62" port: 39893 } } peers { permanent_uuid: "5e792fad2ceb4eb09f6098e4d9996262" member_type: VOTER last_known_addr { host: "127.7.43.61" port: 43381 } } peers { permanent_uuid: "4f7903f02abb445292ada5a8cbc08fd0" member_type: VOTER last_known_addr { host: "127.7.43.60" port: 35287 } } }
I20260810 19:25:23.848914 13343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5fc5bdeec14c20b93b6788546fc8c3 [sys.catalog]: This master's current role is: LEADER
W20260810 19:25:23.903784 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:23.901799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.7.43.61:43381)
0810 19:25:23.902070 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:23.902077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:23.902233 (+ 156us) server_negotiation.cc:407] Connection header received
0810 19:25:23.902463 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:23.902469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:23.902545 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:23.902709 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:23.903540 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37566: BlockingRecv error: recv got EOF from 127.0.0.1:37566 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 19:25:24.168229 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:25:24.166354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.7.43.61:43381)
0810 19:25:24.166571 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:24.166577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:24.166816 (+ 239us) server_negotiation.cc:407] Connection header received
0810 19:25:24.167004 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:24.167008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:24.167083 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:24.167170 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:25:24.167967 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37578: BlockingRecv error: recv got EOF from 127.0.0.1:37578 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 19:25:24.320725 13402 catalog_manager.cc:1259] Loaded cluster ID: e33523c02d1b4f30b34484a3faaa26fb
I20260810 19:25:24.320837 13402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: loading cluster ID for follower catalog manager: success
I20260810 19:25:24.323468 13402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: acquiring CA information for follower catalog manager: success
I20260810 19:25:24.325423 13402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f7903f02abb445292ada5a8cbc08fd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:25:24.509718 13442 catalog_manager.cc:1259] Loaded cluster ID: e33523c02d1b4f30b34484a3faaa26fb
I20260810 19:25:24.509855 13442 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: loading cluster ID for follower catalog manager: success
I20260810 19:25:24.512444 13442 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: acquiring CA information for follower catalog manager: success
I20260810 19:25:24.514333 13442 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e792fad2ceb4eb09f6098e4d9996262: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:25:24.692848 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:24.685226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.7.43.61:43381)
0810 19:25:24.685965 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:24.685971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:24.686329 (+ 358us) server_negotiation.cc:407] Connection header received
0810 19:25:24.686500 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:24.686505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:24.686676 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:24.686814 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:24.687326 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.688453 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.689958 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.690710 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.690848 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:24.691445 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:24.692125 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:24.692293 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:24.692364 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 19:25:24.692457 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":456,"threads_started":1}
I20260810 19:25:24.705739 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:24.699124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.7.43.60:35287)
0810 19:25:24.699727 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:24.699732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:24.699752 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:25:24.699842 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:24.699846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:24.700003 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:24.700140 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:24.700618 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.701414 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.703367 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.703773 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.703866 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:24.704485 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:24.705046 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:24.705187 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:24.705307 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 19:25:24.705391 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":409,"threads_started":1}
I20260810 19:25:24.719985 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:24.712113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.7.43.61:43381)
0810 19:25:24.712311 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:24.712318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:24.712669 (+ 351us) server_negotiation.cc:407] Connection header received
0810 19:25:24.712938 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:24.712944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:24.713039 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:24.713103 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:24.713948 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.715227 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.717568 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.718252 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.718350 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:24.719048 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:24.719201 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:24.719267 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:24.719510 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 19:25:24.719609 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:25:24.727259 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:24.712900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.7.43.60:35287)
0810 19:25:24.713059 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:24.713064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:24.720992 (+ 7928us) server_negotiation.cc:407] Connection header received
0810 19:25:24.721259 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:24.721265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:24.721349 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:24.721410 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:24.722003 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.723147 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.725174 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.725795 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.725891 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:24.726508 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:24.726636 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:24.726693 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:24.726892 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 19:25:24.726969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:25:24.730778 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:25:24.711819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.7.43.62:39893)
0810 19:25:24.724581 (+ 12762us) server_negotiation.cc:206] Beginning negotiation
0810 19:25:24.724587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:25:24.724608 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:25:24.724718 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:25:24.724724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:25:24.724945 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:25:24.725812 (+ 867us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:25:24.725966 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.726961 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.728808 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:25:24.729209 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:25:24.729314 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:25:24.729838 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:25:24.730183 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:25:24.730276 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:25:24.730415 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 19:25:24.730501 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12646,"thread_start_us":12493,"threads_started":1}
I20260810 19:25:24.736245 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 12986
I20260810 19:25:24.737335 13086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:24.849345 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 12986
I20260810 19:25:24.875806 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13105
I20260810 19:25:24.876987 13211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:24.981258 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13105
I20260810 19:25:25.009795 7340 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu pid 13226
I20260810 19:25:25.011040 13317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 19:25:25.118666 7340 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuOTVHW/build/asan/bin/kudu with pid 13226
2026-08-10T19:25:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5125 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9759 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (383109 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST